ASF JIRA
ASF JIRA
Displaying 1000 issues at 19/Mar/20 20:35.
Project Key Summary Issue Type Status Priority Resolution Assignee Reporter Creator Created Last Viewed Updated Resolved Affects Version/s Fix Version/s Component/s Due Date Votes Watchers Images Original Estimate Remaining Estimate Time Spent Work Ratio Sub-Tasks Linked Issues Environment Description Security Level Progress Σ Progress Σ Time Spent Σ Remaining Estimate Σ Original Estimate Labels Git Notification Mailing List Github Integration Git Repository Name Global Rank Git Repository Type Blog Administrator? Blogs - Admin for blog Blogs - Username Blogs - Email Address Docs Text Git Repository Import Path New-TLP-TLPName Blogs - New Blog Write Access Epic Colour Blogs - Existing Blog Name Enable Automatic Patch Review Attachment count Blog - New Blog PMC Epic Name Blog - New Blog Administrators Epic Status Blog - Write access Epic Link Change Category Bug Category Bugzilla - List of usernames Bugzilla - PMC Name Test and Documentation Plan Bugzilla - Email Notification Address Discovered By Blogs - Existing Blog Access Level Complexity Bugzilla - Project Name Severity Initial Confluence Contributors Space Name Space Description Space Key Sprint Rank (Obsolete) Project Machine Readable Info Review Patch? Flags Source Control Link Authors Development Reviewers Ignite Flags Date of First Response Github Integrations - Other Last public comment date Skill Level Affects version (Component) Backport to Version Fix version (Component) Skill Level Existing GitBox Approval Protected Branch GitHub Options Release Note Hadoop Flags Tags Bugzilla Id Level of effort Target Version/s Bug behavior facts Lucene Fields Github Integration - Triggers Workaround Bugzilla Id INFRA - Subversion Repository Path Testcase included Estimated Complexity Regression Review Date Evidence Of Use On World Wide Web Evidence Of Registration Epic/Theme Flagged External issue ID Priority Reproduced In Tags Since Version Reviewer External issue URL Hadoop Flags Issue & fix info Evidence Of Open Source Adoption Rank Severity Tester
ZooKeeper ZOOKEEPER-2756

Add CMake build system for better cross-platform support

Improvement Resolved Major Fixed Andrew Schwartzmeyer Andrew Schwartzmeyer Andrew Schwartzmeyer 13/Apr/17 17:55   10/Aug/17 14:37 10/Aug/17 14:37 3.5.2   build, c client   0 3   Windows and Linux The C bindings primary build system is Autotools. This obviously does not work for Windows, and so the original port to Windows simply added a Visual Studio solution to the project, splitting the build system. As new versions of Visual Studio have come along, new (probably auto-converted) solutions have come along (see zookeeper.sln vs zookeeper-vs2013.sln). When Mesos started being ported to Windows, a Visual Studio 2015 solution was needed, and the previous developer created yet another solution, and setup Mesos' build to patch ZooKeeper and add the 2015 solution. Now Visual Studio 2017 was released, and in the process of moving Mesos ahead, I realized that I would either have to make *yet another* converted solution for ZooKeeper. So instead I tackled the root problem, and ported the Autotools build to CMake, which is a meta-build system which generates files for the in-use platform (whether it be Linux or Solaris or MacOS or Windows).

NOTE: I already have this patch, and will submit it. It has a couple TODOs, and some other things in it that were necessary for Mesos that may need to be pulled into separate patches.
build, windows 9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
2 years, 36 weeks, 5 days ago 0|i3dm4v:
ZooKeeper ZOOKEEPER-2755

Allow to subclass ClientCnxnSocketNetty and NettyServerCnxn in order to use Netty Local transport

New Feature Resolved Major Won't Fix Enrico Olivelli Enrico Olivelli Enrico Olivelli 13/Apr/17 00:51   17/Jan/18 03:02 17/Jan/18 03:02 3.5.2   java client, server   0 3   ClientCnxnSocketNetty and NettyServerCnxn use explicitly InetSocketAddress class to work with network addresses.

We can do a little refactoring to use only SocketAddress and make it possible to create subclasses of ClientCnxnSocketNetty and NettyServerCnxn which leverage built-in Netty 'local' channels.

Such Netty local channels do not create real sockets and so allow a simple ZooKeeper server + ZooKeeper client to be run on the same JVM without binding to real TCP endpoints.

Usecases:

Ability to run concurrently on the same machine tests of projects which use ZooKeeper (usually in unit tests the server and the client run inside the same JVM) without dealing with random ports and in general using less network resources

Run simplified (standalone, all processes in the same JVM) versions of applications which need a working ZooKeeper ensemble to run.

Note:
Embedding ZooKeeper server + client on the same JVM has many risks and in general I think we should encourage users to do so, so I in this patch I will not provide official implementations of ClientCnxnSocketNetty and NettyServerCnxn. There will be implementations only inside the test packages, in order to test that most of the features are working with custom socket factories and in particular with the 'LocalAddress' specific subclass of SocketAddress.

Note:
the 'Local' sockets feature will be available on Netty 4 too
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
2 years, 9 weeks, 1 day ago 0|i3dklr:
ZooKeeper ZOOKEEPER-2754

ZOOKEEPER-3170 Set up Apache Jenkins job that runs the flaky test analyzer script.

Sub-task Resolved Major Fixed Michael Han Michael Han Michael Han 11/Apr/17 18:37   15/Oct/18 06:27 13/Apr/17 17:29   3.4.11, 3.5.4, 3.6.0 tests   0 1   9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
2 years, 49 weeks ago 0|i3didb:
ZooKeeper ZOOKEEPER-2753

ZOOKEEPER-3170 Introduce a python script for generating flaky test report

Sub-task Resolved Major Fixed Michael Han Michael Han Michael Han 11/Apr/17 18:17   15/Oct/18 06:27 15/Oct/18 06:20   3.6.0 tests   0 2   This python script uses Jenkins REST API to query Jenkins build, analyze test results and generate reports of flaky tests over a range of builds across specific time periods.

This is preview of what the dashboard it generates would look like:
http://home.apache.org/~hanm/dashboard/report.html

Pull Request: https://github.com/apache/zookeeper/pull/224
tools 9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
2 years, 12 weeks, 6 days ago 0|i3dicv:
ZooKeeper ZOOKEEPER-2752

ZOOKEEPER-3170 Introduce ZooKeeper flaky test dashboard

Sub-task Open Major Unresolved Michael Han Michael Han Michael Han 11/Apr/17 18:13   29/Aug/19 10:56       tests   0 2   ZooKeeper flaky test dashboard is a set of tools used to track, analyze, and report flaky tests. It's designed with the goal to automate most of the labor intensive part of work related to tracking, monitoring, analyzing and aggregating unit tests results from various Apache ZooKeeper builds. It also increases the visibility of the tests along with (in long term) the over all quality of the ZooKeeper builds. This work is inspired by similar work done on HBase side, with some the tools borrowed from HBase and customized for ZooKeeper use cases.



tools 9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
29 weeks ago 0|i3dicf:
ZooKeeper ZOOKEEPER-2751

Investigate why existing unit tests does not leak connection bean for NIO code path

Task Open Minor Unresolved Unassigned Michael Han Michael Han 10/Apr/17 13:11   10/Apr/17 13:12       server   0 3   In ZOOKEEPER-2743 we observed a race condition in Netty code path that could lead to connection bean leaking; similar code pattern exists in NIO code path as well, however existing unit tests never fail when NIO code path is activated. This is a follow up of ZOOKEEPER-2743 to ensure we don't have connection bean leak when NIO code path is used.

Deliverable:
* Unit tests that fail with connection bean when NIO code path is enabled.
Or
* Proof / analysis that NIO code path does not leak cnx bean and why.
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
2 years, 49 weeks, 3 days ago 0|i3dg13:
ZooKeeper ZOOKEEPER-2750

ZOOKEEPER-3451 Document SSL Support for Atomic Broadcast protocol

Sub-task Closed Major Fixed Andor Molnar Abraham Fine Abraham Fine 07/Apr/17 15:49   01/Jul/19 10:53 11/Mar/19 15:01   3.6.0, 3.5.5     0 4 0 12600   100% 100% 12600 0 pull-request-available, ssl-tls 9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
1 year, 1 week, 3 days ago 0|i3ddin:
ZooKeeper ZOOKEEPER-2749

ZOOKEEPER-2728 Cleanup findbug warnings in branch-3.4: Experimental Warnings

Sub-task Resolved Major Fixed Abraham Fine Abraham Fine Abraham Fine 06/Apr/17 23:49   24/May/17 10:59 08/Apr/17 18:57 3.4.10 3.4.11     0 3   9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
2 years, 49 weeks ago 0|i3dc4n:
ZooKeeper ZOOKEEPER-2748

Admin command to voluntarily drop client connections

New Feature Patch Available Minor Unresolved Marco P. Marco P. Marco P. 06/Apr/17 18:03   30/Jan/19 08:37       server   0 4 0 600   In certain circumstances, it would be useful to be able to move clients from one server to another.

One example: a quorum that consists of 3 servers (A,B,C) with 1000 active client session, where 900 clients are connected to server A, and the remaining 100 are split over B and C (see example below for an example of how this can happen).
A will do a lot more work than B, C.
Overall throughput will benefit by having the clients more evenly divided.
In case of A failure, all its client will create an avalanche by migrating en masse to a different server.

There are other possible use cases for a mechanism to move clients:
- Migrate away all clients before a server restart
- Migrate away part of clients in response to runtime metrics (CPU/Memory usage, ...)
- Shuffle clients after adding more server capacity (i.e. adding Observer nodes)

The simplest form of rebalancing which does not require major changes of protocol or client code consists of requesting a server to voluntarily drop some number of connections.
Clients should be able to transparently move to a different server.

Patch introducing 4-letter commands to shed clients:
https://github.com/apache/zookeeper/pull/215


-- -- --


How client imbalance happens in the first place, an example.

Imagine servers A, B, C and 1000 clients connected.
Initially clients are spread evenly (i.e. 333 clients per server).
A: 333 (restarts: 0)
B: 333 (restarts: 0)
C: 334 (restarts: 0)

Now restart servers a few times, always in A, B, C order (e.g. to pick up a software upgrades or configuration changes).

Restart A:
A: 0 (restarts: 1)
B: 499 (restarts: 0)
C: 500 (restarts: 0)

Restart B:
A: 250 (restarts: 1)
B: 0 (restarts: 1)
C: 750 (restarts: 0)

Restart C:
A: 625 (restarts: 1)
B: 375 (restarts: 1)
C: 0 (restarts: 1)

The imbalance is pretty bad already. C is idle while A has a lot of work.
A second round of restarts makes the situation even worse:

Restart A:
A: 0 (restarts: 2)
B: 688 (restarts: 1)
C: 313 (restarts: 1)

Restart B:
A: 344 (restarts: 2)
B: 657 (restarts: 1)
C: 0 (restarts: 1)

Restart C:
A: 673 (restarts: 2)
B: 328 (restarts: 1)
C: 0 (restarts: 1)

Large cluster (5, 7, 9 servers) make the imbalance even more evident.
100% 100% 600 0 pull-request-available 9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
2 years, 49 weeks, 2 days ago 0|i3dbt3:
ZooKeeper ZOOKEEPER-2747

Fix ZooKeeperAdmin Compilation Warning

Bug Resolved Major Fixed Abraham Fine Abraham Fine Abraham Fine 06/Apr/17 18:01   17/Apr/17 20:42 17/Apr/17 19:58 3.5.2 3.5.4, 3.6.0     0 4   Currently when compiling ZooKeeper we se a compilation warning:
{code}
[javac] /zookeeper/src/java/main/org/apache/zookeeper/admin/ZooKeeperAdmin.java:43: warning: [try] auto-closeable resource ZooKeeperAdmin has a member method close() that could throw InterruptedException
[javac] public class ZooKeeperAdmin extends ZooKeeper {
[javac] ^
[javac] 2 warnings
{code}

This is due to the implementation of AutoCloseable in the ZooKeeper superclass. That class has a warning suppression and explanation, we should copy it to the ZooKeeperAdmin class.
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
2 years, 48 weeks, 2 days ago 0|i3dbsv:
ZooKeeper ZOOKEEPER-2746

Leader hand-off during dynamic reconfig is best effort, while test always expects it

Test Resolved Major Fixed Michael Han Michael Han Michael Han 04/Apr/17 19:39   17/Apr/17 19:48 04/Apr/17 20:03 3.5.2 3.5.4, 3.6.0     0 3   When non-trivial config change happens on the leader (e.g. port change, role change) to minimize disruption of quorum we do leader hand off by having the current leader nominate the next leader instead of kicking off a full leader election.

However this is a best effort and not a guarantee that the new leader of the quorum will be the nominated leader: it is possible that the nominated leader fail to establish the leadership during sync phase, which leads to new election rounds and a different leader elected.

In ReconfigTest.testPortChange one check is the new leader after dynamic reconfig of the current leader has to be a different leader; based on what described earlier this is not always the case as the nominated leader might fail to get a quorum to ack its leadership during sync phase, and the old leader might end up as the new leader. We could either fixed the test by removing the check, or always guarantee that the new leader after dynamic reconfig is the nominated leader (which does not make much sense given the nominated leader also has a possibility to crash.).
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
2 years, 50 weeks, 1 day ago 0|i3d7jr:
ZooKeeper ZOOKEEPER-2745

Node loses data after disk-full event, but successfully joins Quorum

Bug Open Critical Unresolved Unassigned Abhay Bothra Abhay Bothra 04/Apr/17 17:55   21/Nov/18 20:46   3.4.6   server   0 5   Ubuntu 12.04 If disk is full on 1 zookeeper node in a 3 node ensemble, it is able to join the quorum with partial data.

Setup:
--------
- Running a 3 node zookeeper ensemble on Ubuntu 12.04 as upstart services. Let's call the nodes: A, B and C.

Observation:
-----------------
- Connecting to 2 (Node A and B) of the 3 nodes and doing an `ls` in zookeeper data directory was giving:
/foo
/bar
/baz
But an `ls` on node C was giving:
/baz
- On node C, the zookeeper data directory had the following files:
log.1001
log.1600
snapshot.1000 -> size 200
snapshot.1200 -> size 269
snapshot.1300 -> size 300
- Snapshot sizes on node A and B were in the vicinity of 500KB

RCA
-------
- Disk was full on node C prior to the creation time of the small snapshot
files.
- Looking at zookeeper server logs, we observed that zookeeper had crashed and restarted a few times after the first instance of disk full. Everytime time zookeeper starts, it does 3 things:
1. Run the purge task to cleanup old snapshot and txn logs. Our
autopurge.snapRetainCount is set to 3.
2. Restore from the most recent valid snapshot and the txn logs that follow.
3. Take part in a leader election - realize it has missed something - become the follower - get diff of missed txns from the current leader - create a new snapshot of its current state.
- We confirmed that a valid snapshot of the system had existed prior to, and
immediately after the crash. Let's call this snapshot snapshot.800.
- Over the next 3 restarts, zookeeper did the following:
- Purged older snapshots
- Restored from snapshot.800 + txn logs
- Synced up with master, tried to write its updated state to a new snapshot. Crashed due to disk full. The snapshot file, even though invalid, had been created.
- *Note*: This is the first source of the bug. It might be more appropriate to first write the snapshot to a temporary file, and then rename it
snapshot.<txn_id>. That would gives us more confidence in the validity of snapshots in the data dir.
- Let's say the snapshot files created above were snapshot.850, snapshot.920 and snapshot.950
- On the 4th restart, the purge task retained the 3 recent snapshots - snapshot.850, snapshot.920, and snapshot.950, and proceeded to purge snapshot.800 and associated txn logs assuming that they were no longer needed.
- *Note*: This is the second source of the bug. Instead of retaining the 3 most recent *valid* snapshots, the server just retains 3 most recent snapshots, regardless of their validity.
- When restoring, zookeeper doesn't find any valid snapshot logs to restore from. So it tries to reload its state from txn logs starting at zxid 0. However, those transaction logs would have long ago been garbage collected. It reloads from whatever txn logs are present. Let's say the only txn log file present (log.951) contains logs for zxid 951 to 998. It reloads from that log file, syncs with master - gets txns 999 and 1000, and writes the snapshot log snapshot.1000 to disk. Now that we have deleted snapshot.800, we have enough free disk space to write snapshot.1000. From this state onwards, zookeeper will always assume it has the state till txn id 1000, even though it only has state from txn id 951 to 1000.
9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
1 year, 17 weeks ago 0|i3d7dr:
ZooKeeper ZOOKEEPER-2744

Typos in the comments of ZooKeeper class

Improvement Resolved Trivial Fixed Abraham Fine Ethan Li Ethan Li 02/Apr/17 22:56   17/Apr/17 20:42 17/Apr/17 20:03 3.4.10, 3.5.2 3.5.4, 3.6.0     0 5   In the comments of ZooKeeper class definition, "This special event has type EventNone and state sKeeperStateDisconnected." should be "This special event has EventType None and KeeperState Disconnected." 9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
2 years, 48 weeks, 2 days ago 0|i3d433:
ZooKeeper ZOOKEEPER-2743

Netty connection leaks JMX connection bean upon connection close in certain race conditions.

Bug Resolved Major Fixed Michael Han Michael Han Michael Han 01/Apr/17 16:30   17/Apr/17 19:44 10/Apr/17 13:20 3.4.10, 3.5.2 3.4.11, 3.5.4, 3.6.0 server   0 4   This is a tricky issue found while debugging failure of "flaky" watcher test (ZOOKEEPER-2686). When closing a Netty connection, depend on timing the connection bean registered when the connection was provisioned might not get unregistered, leading to leaked Java beans.

The race happens at the time when the client is in the process of finalizing the session. As part of session finalization, a connection bean will be registered [1]. But right before the registering bean, the connection might gets closed, in cases for example the server that the client is connecting to is shutdown. As part of connection close, the bean will be un-registered, as expected [2], however the problem is when we execute at [2], the connection bean might not finish registering at [1], so the unregister of bean is a NOP. What's worse, as part of connection close, we remove this connection from connection factory [3], so future connection close call will get short circuited and directly return; in other words the bean unregister code in connection close call will only get executed once. Depends on luck, the bean might not get unregistered, as previously illustrated.


[1] https://github.com/apache/zookeeper/blob/master/src/java/main/org/apache/zookeeper/server/ZooKeeperServer.java#L700
[2] https://github.com/apache/zookeeper/blob/master/src/java/main/org/apache/zookeeper/server/NettyServerCnxn.java#L114
[3]
https://github.com/apache/zookeeper/blob/master/src/java/main/org/apache/zookeeper/server/NettyServerCnxn.java#L96
netty 9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
2 years, 49 weeks, 3 days ago 0|i3d3l3:
ZooKeeper ZOOKEEPER-2742

Few test cases of org.apache.zookeeper.ZooKeeperTest fails in Windows

Test Resolved Trivial Fixed Abhishek Kumar Abhishek Kumar Abhishek Kumar 30/Mar/17 11:08   04/May/17 06:11 29/Apr/17 14:20   3.5.4, 3.6.0 tests   0 4   Windows Following test cases fail in Windows environment:
1. org.apache.zookeeper.ZooKeeperTest.testLsrRootCommand()
2. org.apache.zookeeper.ZooKeeperTest.testLsrCommand()

It seems that failure is related to use of "\n" (System dependent new line char)in org.apache.zookeeper.ZooKeeperTest.runCommandExpect(CliCommand, List<String>)

......................................
......................................
String result = byteStream.toString();
assertTrue(result, result.contains(
StringUtils.joinStrings(expectedResults, "\n")));
......................................
......................................
9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
2 years, 46 weeks, 5 days ago 0|i3d093:
ZooKeeper ZOOKEEPER-2741

A Swift Connector of ZooKeeper is available now: Perfect-ZooKeeper

New Feature Open Minor Unresolved Unassigned Rockford Wei Rockford Wei 28/Mar/17 14:52   28/Mar/17 14:54       contrib   0 1   Perfect-ZooKeeper is a Swift class wrapper of zookeeper C connector:

Source Code:
https://github.com/PerfectlySoft/Perfect-ZooKeeper

Document:
http://www.perfect.org/docs/ZooKeeper.html

Perfect is an open source Server Side Swift framework supported by PerfectlySoft Inc. since 2015.

We are happy to share this new components as a community contribution.
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
2 years, 51 weeks, 2 days ago 0|i3cwpr:
ZooKeeper ZOOKEEPER-2740

Port ZOOKEEPER-2737 to branch-3.4

Bug Resolved Critical Fixed Michael Han Michael Han Michael Han 27/Mar/17 13:35   20/Jul/17 13:42 20/Jul/17 13:42 3.4.9, 3.4.10 3.4.11 server   0 2   ZOOKEEPER-2737 fix is pending to branch-3.4 because we are in middle of release. This fix should get in after 3.4.10 gets out.

cc [~rakeshr].
netty 9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
2 years, 35 weeks ago 0|i3cuk7:
ZooKeeper ZOOKEEPER-2739

maxClientCnxns not working in NettyServerCnxnFactory

Bug Open Major Unresolved Unassigned Vincent Poon Vincent Poon 24/Mar/17 21:22   25/Mar/17 00:34   3.4.9, 3.5.0, 3.6.0       0 4   The maxClientCnxns field isn't being used in NettyServerCnxnFactory, and therefore the connection limit isn't observed.

See attached test
9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
2 years, 51 weeks, 5 days ago 0|i3cs9j:
ZooKeeper ZOOKEEPER-2738

maxClientCnxns not limiting concurrent connections properly

Bug Patch Available Major Unresolved Unassigned Vincent Poon Vincent Poon 24/Mar/17 21:12   30/Jan/19 08:33   3.5.2, 3.6.0       0 4 0 600   The test MaxCnxnsTest is incorrect as it only creates up the maxCnxns number of threads, whereas it should create more. See attached patch

When the test is fixed, it fails on master and 3.5, where ZOOKEEPER-1504 removed some synchronization.
100% 100% 600 0 pull-request-available 9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
2 years, 51 weeks, 2 days ago 0|i3cs93:
ZooKeeper ZOOKEEPER-2737

NettyServerCnxFactory leaks connection if exception happens while writing to a channel.

Bug Closed Critical Fixed Michael Han Michael Han Michael Han 24/Mar/17 13:16   17/May/17 23:43 27/Mar/17 13:32 3.5.2 3.5.3, 3.6.0 server   0 4   Found this while debugging occasionally failed unit tests. Currently we do this if exception occurs during writing to a channel with Netty:

{code}
@Override
public void exceptionCaught(ChannelHandlerContext ctx, ExceptionEvent e)
throws Exception
{
LOG.warn("Exception caught " + e, e.getCause());
NettyServerCnxn cnxn = (NettyServerCnxn) ctx.getAttachment();
if (cnxn != null) {
if (LOG.isDebugEnabled()) {
LOG.debug("Closing " + cnxn);
cnxn.close();
}
}
}
{code}

So the connection is only closed when debug mode is enabled. This is problematic as lots of clean up code is abstracted inside the close and without proper close the connection we are leaking resources.

[Commit log|https://github.com/apache/zookeeper/blob/master/src/java/main/org/apache/zookeeper/server/NettyServerCnxnFactory.java#L147] indicates the issue exists since day 1 with ZOOKEEPER-733. Note the original patch uploaded to ZOOKEEPER-733 has this close call in right place, and the call gets moved around during iteration of the patches w/o gets noticed.
connection, netty, server 9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
2 years, 51 weeks, 3 days ago 0|i3cre7:
ZooKeeper ZOOKEEPER-2736

Add a connection rate limiter

Improvement Resolved Major Duplicate Unassigned Vincent Poon Vincent Poon 23/Mar/17 14:30   08/Aug/19 13:14 08/Aug/19 13:13 3.4.9, 3.5.2   server   0 7 0 1200   Currently the maxClientCnxns property only limits the aggregate number of connections from a client, but not the rate at which connections can be created.

This patch adds a configurable connection rate limiter which limits the rate as well.
100% 100% 1200 0 pull-request-available 9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
2 years, 50 weeks, 3 days ago 0|i3cpt3:
ZooKeeper ZOOKEEPER-2735

Typo fixes in some scripts

Bug Resolved Trivial Fixed Woojin Joe Woojin Joe Woojin Joe 22/Mar/17 21:27   12/Jul/17 23:10 23/Mar/17 13:22   3.6.0 scripts   0 5   There are same typos in shell files follows:
* https://github.com/apache/zookeeper/blob/master/bin/zkCleanup.sh#L28
* https://github.com/apache/zookeeper/blob/master/bin/zkCli.sh#L28
* https://github.com/apache/zookeeper/blob/master/bin/zkServer.sh#L25

TO-BE
need to be changed from *POSTIX* to *POSIX*
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years ago 0|i3co6v:
ZooKeeper ZOOKEEPER-2734

3.5.3 should be a beta release instead of alpha release.

Task Closed Blocker Fixed Michael Han Michael Han Michael Han 21/Mar/17 13:52   17/May/17 23:44 27/Mar/17 23:49 3.5.2 3.5.3 build   0 2   Currently 3.5.3 is tagged as alpha both in build and JIRA. We should reach a consensus on the tag before release. See the email thread on dev list for more details.

Deliverable:
* A consensus on using beta as the version for 3.5.3 release.
* Update build.xml.
* Update JIRA (3.5.3-alpha -> 3.5.3-beta).

Thread on dev list:
http://mail-archives.apache.org/mod_mbox/zookeeper-dev/201703.mbox/%3CCA%2Bi0x1JZacVMQGd_Jb34jSaw7p_nhWpcx9uzHwCwPPF%2BPPrf3g%40mail.gmail.com%3E
build, release 9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
2 years, 51 weeks, 2 days ago 0|i3clbj:
ZooKeeper ZOOKEEPER-2733

ZOOKEEPER-2728 Cleanup findbug warnings in branch-3.4: Dodgy code Warnings

Sub-task Resolved Major Fixed Abraham Fine Rakesh Radhakrishnan Rakesh Radhakrishnan 20/Mar/17 01:29   21/Jun/17 19:26 24/May/17 10:57 3.4.10 3.4.11     0 4   Please refer the attached sheet in parent jira. Below is the details of findbug warnings.
{code}
DB org.apache.zookeeper.server.quorum.auth.SaslQuorumAuthLearner.send(DataOutputStream, byte[]) uses the same code for two branches
DLS Dead store to txn in org.apache.zookeeper.server.quorum.LearnerHandler.packetToString(QuorumPacket)
NP Load of known null value in org.apache.zookeeper.server.PrepRequestProcessor.pRequest(Request)
NP Possible null pointer dereference in org.apache.zookeeper.server.PurgeTxnLog.purgeOlderSnapshots(FileTxnSnapLog, File) due to return value of called method
NP Possible null pointer dereference in org.apache.zookeeper.server.PurgeTxnLog.purgeOlderSnapshots(FileTxnSnapLog, File) due to return value of called method
NP Load of known null value in org.apache.zookeeper.server.quorum.auth.SaslQuorumAuthLearner.send(DataOutputStream, byte[])
NP Load of known null value in org.apache.zookeeper.server.quorum.auth.SaslQuorumAuthServer.send(DataOutputStream, byte[], QuorumAuth$Status)
NP Possible null pointer dereference in org.apache.zookeeper.server.upgrade.UpgradeMain.copyFiles(File, File, String) due to return value of called method
RCN Redundant nullcheck of bytes, which is known to be non-null in org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.next()
SF Switch statement found in org.apache.zookeeper.server.PrepRequestProcessor.pRequest(Request) where default case is missing
SF Switch statement found in org.apache.zookeeper.server.PrepRequestProcessor.pRequest2Txn(int, long, Request, Record, boolean) where default case is missing
SF Switch statement found in org.apache.zookeeper.server.quorum.AuthFastLeaderElection$Messenger$WorkerReceiver.run() where default case is missing
SF Switch statement found in org.apache.zookeeper.server.quorum.AuthFastLeaderElection$Messenger$WorkerSender.process(AuthFastLeaderElection$ToSend) where default case is missing
SF Switch statement found in org.apache.zookeeper.server.quorum.Follower.processPacket(QuorumPacket) where default case is missing
SF Switch statement found in org.apache.zookeeper.server.quorum.Observer.processPacket(QuorumPacket) where default case is missing
ST Write to static field org.apache.zookeeper.server.SyncRequestProcessor.randRoll from instance method org.apache.zookeeper.server.SyncRequestProcessor.run()
UrF Unread public/protected field: org.apache.zookeeper.server.upgrade.DataTreeV1$ProcessTxnResult.err
UrF Unread public/protected field: org.apache.zookeeper.server.upgrade.DataTreeV1$ProcessTxnResult.path
UrF Unread public/protected field: org.apache.zookeeper.server.upgrade.DataTreeV1$ProcessTxnResult.stat
UrF Unread public/protected field: org.apache.zookeeper.server.upgrade.DataTreeV1$ProcessTxnResult.type
{code}
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
2 years, 39 weeks, 1 day ago 0|i3chzj:
ZooKeeper ZOOKEEPER-2732

ZOOKEEPER-2728 Cleanup findbug warnings in branch-3.4: Performance Warnings

Sub-task Resolved Major Fixed Abraham Fine Rakesh Radhakrishnan Rakesh Radhakrishnan 20/Mar/17 01:25   22/May/17 13:58 22/May/17 00:26   3.4.11     0 2   Please refer the attached sheet in parent jira. Below is the details of findbug warnings.
{code}
Bx Boxing/unboxing to parse a primitive new org.apache.zookeeper.server.quorum.QuorumCnxManager(long, Map, QuorumAuthServer, QuorumAuthLearner, int, boolean, int, boolean)
Bx new org.apache.zookeeper.server.quorum.QuorumCnxManager(long, Map, QuorumAuthServer, QuorumAuthLearner, int, boolean, int, boolean) invokes inefficient new Integer(String) constructor; use Integer.valueOf(String) instead
Dm org.apache.zookeeper.server.quorum.FastLeaderElection$Notification.toString() invokes inefficient new String(String) constructor
WMI org.apache.zookeeper.server.DataTree.dumpEphemerals(PrintWriter) makes inefficient use of keySet iterator instead of entrySet iterator
WMI org.apache.zookeeper.server.quorum.flexible.QuorumHierarchical.computeGroupWeight() makes inefficient use of keySet iterator instead of entrySet iterator
WMI org.apache.zookeeper.server.quorum.flexible.QuorumHierarchical.containsQuorum(HashSet) makes inefficient use of keySet iterator instead of entrySet iterator
WMI org.apache.zookeeper.ZooKeeperMain.usage() makes inefficient use of keySet iterator instead of entrySet iterator
{code}
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
2 years, 43 weeks, 3 days ago 0|i3chzb:
ZooKeeper ZOOKEEPER-2731

ZOOKEEPER-2728 Cleanup findbug warnings in branch-3.4: Malicious code vulnerability Warnings

Sub-task Resolved Major Fixed Abraham Fine Rakesh Radhakrishnan Rakesh Radhakrishnan 20/Mar/17 01:22   24/May/17 12:59 24/May/17 12:17 3.4.9 3.4.11     0 2   Please refer the attached sheet in parent jira. Below is the details of findbug warnings.
{code}
MS org.apache.zookeeper.Environment.JAAS_CONF_KEY isn't final but should be
Bug type MS_SHOULD_BE_FINAL (click for details)
In class org.apache.zookeeper.Environment
Field org.apache.zookeeper.Environment.JAAS_CONF_KEY
At Environment.java:[line 34]

MS org.apache.zookeeper.server.ServerCnxn.cmd2String is a mutable collection which should be package protected
Bug type MS_MUTABLE_COLLECTION_PKGPROTECT (click for details)
In class org.apache.zookeeper.server.ServerCnxn
Field org.apache.zookeeper.server.ServerCnxn.cmd2String
At ServerCnxn.java:[line 230]

MS org.apache.zookeeper.ZooDefs$Ids.OPEN_ACL_UNSAFE is a mutable collection
Bug type MS_MUTABLE_COLLECTION (click for details)
In class org.apache.zookeeper.ZooDefs$Ids
Field org.apache.zookeeper.ZooDefs$Ids.OPEN_ACL_UNSAFE
At ZooDefs.java:[line 100]

MS org.apache.zookeeper.ZooKeeperMain.commandMap is a mutable collection which should be package protected
Bug type MS_MUTABLE_COLLECTION_PKGPROTECT (click for details)
In class org.apache.zookeeper.ZooKeeperMain
Field org.apache.zookeeper.ZooKeeperMain.commandMap
At ZooKeeperMain.java:[line 53]
{code}
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
2 years, 43 weeks, 1 day ago 0|i3chz3:
ZooKeeper ZOOKEEPER-2730

ZOOKEEPER-2728 Cleanup findbug warnings in branch-3.4: Disable Internationalization Warnings

Sub-task Resolved Major Fixed Janek P Rakesh Radhakrishnan Rakesh Radhakrishnan 20/Mar/17 01:19   19/Jun/17 09:27 19/Jun/17 06:29   3.4.11     0 4   Please refer the attached sheet in parent jira. Below is the details of findbug warnings.
{code}
Dm Found reliance on default encoding in org.apache.jute.compiler.CGenerator.genCode(): new java.io.FileWriter(File)
Dm Found reliance on default encoding in org.apache.jute.compiler.CppGenerator.genCode(): new java.io.FileWriter(File)
Dm Found reliance on default encoding in org.apache.jute.compiler.JRecord.genCsharpCode(File): new java.io.FileWriter(File)
Dm Found reliance on default encoding in org.apache.jute.compiler.JRecord.genJavaCode(File): new java.io.FileWriter(File)
Dm Found reliance on default encoding in new org.apache.jute.XmlOutputArchive(OutputStream): new java.io.PrintStream(OutputStream)
Dm Found reliance on default encoding in org.apache.zookeeper.client.FourLetterWordMain.send4LetterWord(String, int, String, int): new java.io.InputStreamReader(InputStream)
Dm Found reliance on default encoding in org.apache.zookeeper.client.FourLetterWordMain.send4LetterWord(String, int, String, int): String.getBytes()
Dm Found reliance on default encoding in org.apache.zookeeper.ClientCnxn$SendThread.pingRwServer(): new java.io.InputStreamReader(InputStream)
Dm Found reliance on default encoding in org.apache.zookeeper.ClientCnxn$SendThread.pingRwServer(): String.getBytes()
Dm Found reliance on default encoding in org.apache.zookeeper.server.auth.DigestAuthenticationProvider.generateDigest(String): String.getBytes()
Dm Found reliance on default encoding in org.apache.zookeeper.server.auth.DigestAuthenticationProvider.handleAuthentication(ServerCnxn, byte[]): new String(byte[])
Dm Found reliance on default encoding in org.apache.zookeeper.server.DataTree.updateBytes(String, long): new String(byte[])
Dm Found reliance on default encoding in org.apache.zookeeper.server.DataTree.updateBytes(String, long): String.getBytes()
Dm Found reliance on default encoding in org.apache.zookeeper.server.DataTree.updateCount(String, int): new String(byte[])
Dm Found reliance on default encoding in org.apache.zookeeper.server.DataTree.updateCount(String, int): String.getBytes()
Dm Found reliance on default encoding in org.apache.zookeeper.server.DataTree.updateQuotaForPath(String): String.getBytes()
Dm Found reliance on default encoding in org.apache.zookeeper.server.NettyServerCnxn$SendBufferWriter.checkFlush(boolean): String.getBytes()
Dm Found reliance on default encoding in org.apache.zookeeper.server.NIOServerCnxn$SendBufferWriter.checkFlush(boolean): String.getBytes()
Dm Found reliance on default encoding in org.apache.zookeeper.server.persistence.FileSnap.<static initializer for FileSnap>(): String.getBytes()
Dm Found reliance on default encoding in org.apache.zookeeper.server.persistence.FileTxnLog.<static initializer for FileTxnLog>(): String.getBytes()
Dm Found reliance on default encoding in org.apache.zookeeper.server.quorum.QuorumPeer.readLongFromFile(String): new java.io.FileReader(File)
Dm Found reliance on default encoding in org.apache.zookeeper.server.quorum.QuorumPeer.writeLongToFile(String, long): new java.io.OutputStreamWriter(OutputStream)
Dm Found reliance on default encoding in org.apache.zookeeper.server.quorum.QuorumPeerConfig.parseProperties(Properties): new java.io.FileReader(File)
Dm Found reliance on default encoding in org.apache.zookeeper.server.Request.toString(): new String(byte[])
Dm Found reliance on default encoding in org.apache.zookeeper.server.ServerCnxn.<static initializer for ServerCnxn>(): String.getBytes()
Dm Found reliance on default encoding in org.apache.zookeeper.server.TraceFormatter.main(String[]): new String(byte[])
Dm Found reliance on default encoding in org.apache.zookeeper.server.util.OSMXBean.getMaxFileDescriptorCount(): new java.io.InputStreamReader(InputStream)
Dm Found reliance on default encoding in org.apache.zookeeper.server.util.OSMXBean.getOpenFileDescriptorCount(): new java.io.InputStreamReader(InputStream)
Dm Found reliance on default encoding in org.apache.zookeeper.ServerAdminClient.dump(String, int): new String(byte[])
Dm Found reliance on default encoding in org.apache.zookeeper.ServerAdminClient.dump(String, int): String.getBytes()
Dm Found reliance on default encoding in org.apache.zookeeper.ServerAdminClient.getTraceMask(String, int): String.getBytes()
Dm Found reliance on default encoding in org.apache.zookeeper.ServerAdminClient.kill(String, int): new String(byte[])
Dm Found reliance on default encoding in org.apache.zookeeper.ServerAdminClient.kill(String, int): String.getBytes()
Dm Found reliance on default encoding in org.apache.zookeeper.ServerAdminClient.ruok(String, int): new String(byte[])
Dm Found reliance on default encoding in org.apache.zookeeper.ServerAdminClient.ruok(String, int): String.getBytes()
Dm Found reliance on default encoding in org.apache.zookeeper.ServerAdminClient.setTraceMask(String, int, String): String.getBytes()
Dm Found reliance on default encoding in org.apache.zookeeper.ServerAdminClient.stat(String, int): new String(byte[])
Dm Found reliance on default encoding in org.apache.zookeeper.ServerAdminClient.stat(String, int): String.getBytes()
Dm Found reliance on default encoding in org.apache.zookeeper.Shell.runCommand(): new java.io.InputStreamReader(InputStream)
Dm Found reliance on default encoding in org.apache.zookeeper.version.util.VerGen.generateFile(File, VerGen$Version, String, String): new java.io.FileWriter(File)
Dm Found reliance on default encoding in org.apache.zookeeper.ZooKeeperMain.createQuota(ZooKeeper, String, long, int): new String(byte[])
Dm Found reliance on default encoding in org.apache.zookeeper.ZooKeeperMain.createQuota(ZooKeeper, String, long, int): String.getBytes()
Dm Found reliance on default encoding in org.apache.zookeeper.ZooKeeperMain.delQuota(ZooKeeper, String, boolean, boolean): new String(byte[])
Dm Found reliance on default encoding in org.apache.zookeeper.ZooKeeperMain.delQuota(ZooKeeper, String, boolean, boolean): String.getBytes()
Dm Found reliance on default encoding in org.apache.zookeeper.ZooKeeperMain.processZKCmd(ZooKeeperMain$MyCommandOptions): new String(byte[])
Dm Found reliance on default encoding in org.apache.zookeeper.ZooKeeperMain.processZKCmd(ZooKeeperMain$MyCommandOptions): String.getBytes()
Dm Found reliance on default encoding in org.apache.zookeeper.ZooKeeperMain.run(): new java.io.InputStreamReader(InputStream)
Dm Found reliance on default encoding in org.apache.zookeeper.ZooKeeperMain$1.processResult(int, String, Object, byte[], Stat): new String(byte[])
{code}
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
2 years, 39 weeks, 3 days ago 0|i3chyv:
ZooKeeper ZOOKEEPER-2729

ZOOKEEPER-2728 Cleanup findbug warnings in branch-3.4: Correctness Warnings

Sub-task Resolved Major Fixed Abraham Fine Rakesh Radhakrishnan Rakesh Radhakrishnan 20/Mar/17 01:16   13/Apr/17 12:57 08/Apr/17 18:31 3.4.9 3.4.11     0 3   {code}
(1) INT Bad comparison of nonnegative value with 0 in org.apache.zookeeper.server.quorum.auth.SaslQuorumAuthLearner.send(DataOutputStream, byte[])
Bug type INT_BAD_COMPARISON_WITH_NONNEGATIVE_VALUE (click for details)
In class org.apache.zookeeper.server.quorum.auth.SaslQuorumAuthLearner
In method org.apache.zookeeper.server.quorum.auth.SaslQuorumAuthLearner.send(DataOutputStream, byte[])
Value 0
At SaslQuorumAuthLearner.java:[line 176]

(2) INT Bad comparison of nonnegative value with 0 in org.apache.zookeeper.server.quorum.auth.SaslQuorumAuthServer.send(DataOutputStream, byte[], QuorumAuth$Status)
Bug type INT_BAD_COMPARISON_WITH_NONNEGATIVE_VALUE (click for details)
In class org.apache.zookeeper.server.quorum.auth.SaslQuorumAuthServer
In method org.apache.zookeeper.server.quorum.auth.SaslQuorumAuthServer.send(DataOutputStream, byte[], QuorumAuth$Status)
Value 0
At SaslQuorumAuthServer.java:[line 170]
{code}
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
2 years, 49 weeks ago 0|i3chyn:
ZooKeeper ZOOKEEPER-2728

Clean up findbug warnings in branch-3.4

Bug Resolved Major Fixed Rakesh Radhakrishnan Rakesh Radhakrishnan Rakesh Radhakrishnan 20/Mar/17 01:13   12/Jul/17 23:04 19/Jun/17 06:36 3.4.9 3.4.11     1 3   ZOOKEEPER-2729, ZOOKEEPER-2730, ZOOKEEPER-2731, ZOOKEEPER-2732, ZOOKEEPER-2733, ZOOKEEPER-2749, ZOOKEEPER-2762 This jira to cleanup findbug warnings reported in branch-3.4

[Branch3.4 FindbugsWarnings.html|https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/444/artifact/build/test/findbugs/newPatchFindbugsWarnings.html]
9223372036854775807 No Perforce job exists for this issue. 2 9223372036854775807
2 years, 39 weeks, 3 days ago 0|i3chyf:
ZooKeeper ZOOKEEPER-2727

WARN and stacktrace for normally closed socket

Bug Resolved Major Won't Fix Mark Fenes Andrey Andrey 17/Mar/17 11:48   23/Mar/18 10:43 23/Mar/18 10:43 3.4.9       0 4   Steps to reproduce:
* setup zookeeper
* setup TCP load balancer. This balancer should check zookeeper clientPort liveness(healthcheck) by opening and closing TCP connection to clientPort. See https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/ or https://www.digitalocean.com/community/tutorials/how-to-create-your-first-digitalocean-load-balancer#step-2-—-creating-the-load-balancer for details.
* in logs:
{code}
2017-03-17 15:41:19,843 [myid:1] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@357] - caught end of stream exception
EndOfStreamException: Unable to read additional data from client sessionid 0x0, likely client has closed socket
at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
at java.lang.Thread.run(Thread.java:745)
{code}

Issue is here:
https://github.com/apache/zookeeper/blob/5fe68506f217246c7ebd96803f9c78e13ec2f11a/src/java/main/org/apache/zookeeper/server/NIOServerCnxn.java#L322

-1 is a normal socket termination.

Expected:
* reduce log level to INFO
* do not log stacktrace.
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
1 year, 51 weeks, 6 days ago 0|i3cfm7:
ZooKeeper ZOOKEEPER-2726

Patch for ZOOKEEPER-2693 introduces potential race condition

Bug Closed Major Fixed Kyle Nusbaum Kyle Nusbaum Kyle Nusbaum 16/Mar/17 15:15   31/Mar/17 05:01 16/Mar/17 16:01   3.4.10, 3.5.3, 3.6.0     0 5   We noticed when porting the patch, that isEnabled is not thread-safe.
Synchronizing it and resetWhitelist should solve the issue.
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 6 days ago 0|i3cdvr:
ZooKeeper ZOOKEEPER-2725

Upgrading to a global session fails with a multiop

Bug Resolved Major Fixed Brian Nixon Brian Nixon Brian Nixon 15/Mar/17 18:34   12/Jul/17 23:05 21/Mar/17 12:50 3.5.2 3.5.4, 3.6.0 server   0 5   On an ensemble with local sessions enabled, when a client with a local session requests the creation of an ephemeral node within a multi-op, the client gets a session expired message. The same multi-op works if the session is already global. This breaks the client's expectation of seamless promotion from local session to global session server-side. 9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 2 days ago 0|i3cc13:
ZooKeeper ZOOKEEPER-2724

Skip cert files for releaseaudit target.

Improvement Closed Blocker Fixed Michael Han Michael Han Michael Han 15/Mar/17 17:21   17/May/17 23:44 17/Mar/17 10:59 3.5.2 3.5.3 build   0 2   In branch-3.5 release auditing generating warnings against cert files as these files don't contain Apache License (AL) header. I don't think these files should be checked because they are not source files, and we skip them in master branch. We should do the same for branch-3.5 by skipping these cert files as well. This should be fixed before 3.5.3 release.

Attach the snippet of warning for reference:
{noformat}
[rat:report] !????? /home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-github-pr-build/build/zookeeper-3.5.3-alpha-SNAPSHOT/contrib/rest/conf/keys/rest.cer
[rat:report] !????? /home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-github-pr-build/build/zookeeper-3.5.3-alpha-SNAPSHOT/src/contrib/rest/conf/keys/rest.cer
{noformat}
build 9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 6 days ago 0|i3cbwf:
ZooKeeper ZOOKEEPER-2723

ConnectStringParser does not parse correctly if quorum string has znode path

Bug Open Major Unresolved Vishal Khandelwal Vishal Khandelwal Vishal Khandelwal 15/Mar/17 07:11   18/Apr/17 16:45           0 3   f2017-03-14 07:10:26,247 INFO [main] zookeeper.ZooKeeper - Initiating client connection, connectString=x1-1-was.ops.sfdc.net:2181,x2-1-was.ops.sfdc.net:2181,x3-1-was.ops.sfdc.net:2181,x4-1-was.ops.sfdc.net:2181,x5-1-was.ops.sfdc.net:2181:/hbase sessionTimeout=60000 watcher=org.apache.hadoop.hbase.zookeeper.PendingWatcher@6e16b8b5 2017-03-14 07:10:26,250 ERROR [main] client.StaticHostProvider - Unable to connect to server: x5-1-was.ops.sfdc.net:2181:2181 java.net.UnknownHostException: x5-1-was.ops.sfdc.net:2181: Name or service not known at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method) at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928) at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323) at java.net.InetAddress.getAllByName0(InetAddress.java:1276) at java.net.InetAddress.getAllByName(InetAddress.java:1192) at java.net.InetAddress.getAllByName(InetAddress.java:1126) at org.apache.zookeeper.client.StaticHostProvider.<init>(StaticHostProvider.java:60) at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:446) at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:380) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.checkZk(RecoverableZooKeeper.java:141) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.<init>(RecoverableZooKeeper.java:128) at org.apache.hadoop.hbase.zookeeper.ZKUtil.connect(ZKUtil.java:135) at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:173) at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:147) at org.apache.hadoop.hbase.client.ZooKeeperKeepAliveConnection.<init>(ZooKeeperKeepAliveConnection.java:43) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getKeepAliveZooKeeperWatcher(HConnectionManager.java:1875) at org.apache.hadoop.hbase.client.ZooKeeperRegistry.getClusterId(ZooKeeperRegistry.java:82) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.retrieveClusterId(HConnectionManager.java:929) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.<init>(HConnectionManager.java:714) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.client.HConnectionManager.createConnection(HConnectionManager.java:466) at org.apache.hadoop.hbase.client.HConnectionManager.createConnection(HConnectionManager.java:445) at org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnectionManager.java:326) 9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
2 years, 48 weeks, 2 days ago 0|i3caqv:
ZooKeeper ZOOKEEPER-2722

Flaky Test: org.apache.zookeeper.test.ReadOnlyModeTest.testSessionEstablishment

Bug Resolved Major Fixed Michael Han Michael Han Michael Han 14/Mar/17 19:16   24/Apr/17 12:55 17/Apr/17 19:38   3.4.11, 3.5.4, 3.6.0 tests   0 5   {noformat}
Error Message

KeeperErrorCode = ConnectionLoss for /test
Stacktrace

org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /test
at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:1423)
at org.apache.zookeeper.test.ReadOnlyModeTest.testSessionEstablishment(ReadOnlyModeTest.java:238)
at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:79)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.lang.Thread.run(Thread.java:745)
{noformat}

Looks like we should retry before giving up.
flaky 9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
2 years, 47 weeks, 3 days ago 1 1 0|i3c9xr:
ZooKeeper ZOOKEEPER-2721

org.apache.zookeeper.server.quorum.ReconfigRecoveryTest fails intermittently

Test Open Major Unresolved Unassigned Sneha Kanekar Sneha Kanekar 14/Mar/17 05:31   14/Mar/17 05:48   3.6.0   quorum, server, tests   0 2   Ubuntu:14.04 The test-suite org.apache.zookeeper.server.quorum.ReconfigRecoveryTest fails intermittently on ppc64le and x86 architechture. I have attached standard output log. The error message is as follows:
{code:borderStyle=solid}
Testcase: testCurrentServersAreObserversInNextConfig took 90.488 sec
FAILED
waiting for server 3 being up
junit.framework.AssertionFailedError: waiting for server 3 being up
at org.apache.zookeeper.server.quorum.ReconfigRecoveryTest.testCurrentServersAreObserversInNextConfig(ReconfigRecoveryTest.java:217)
at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:79)
{code}

Also this issue is related to ZOOKEEPER-1806 and ZOOKEEPER-2080. Both of them are marked as fixed and still i am getting this failure.
ppc64le, x86 9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
3 years, 1 week, 2 days ago 0|i3c8fb:
ZooKeeper ZOOKEEPER-2720

org.apache.zookeeper.test.WatchEventWhenAutoResetTest fails intermittently

Test Resolved Major Duplicate Michael Han Sneha Kanekar Sneha Kanekar 14/Mar/17 02:05   24/Aug/17 12:49 24/Aug/17 12:49 3.6.0   tests   0 3   Ubuntu:14.04 The test-suite org.apache.zookeeper.test.WatchEventWhenAutoResetTest fails intermittently. It is failing on ppc64le and x86 architechture. I have attached standard output log. The error message is as follows:

{code:borderStyle=solid}
Testcase: testNodeDataChanged took 1.959 sec
FAILED
expected:<NodeDataChanged> but was:<NodeDeleted>
junit.framework.AssertionFailedError: expected:<NodeDataChanged> but was:<NodeDeleted>
at org.apache.zookeeper.test.WatchEventWhenAutoResetTest$EventsWatcher.assertEvent(WatchEventWhenAutoResetTest.java:67)
at org.apache.zookeeper.test.WatchEventWhenAutoResetTest.testNodeDataChanged(WatchEventWhenAutoResetTest.java:117)
at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:79)
{code}
ppc64le, x86 9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
2 years, 30 weeks, 1 day ago 0|i3c83r:
ZooKeeper ZOOKEEPER-2719

Port ZOOKEEPER-2169 (TTL Nodes) to 3.5 branch

New Feature Closed Major Fixed Jordan Zimmerman Jordan Zimmerman Jordan Zimmerman 12/Mar/17 13:38   21/Jan/19 08:41 17/Mar/17 11:05   3.5.3 java client, server   3 4   ZOOKEEPER-2169 is a useful feature that should be deployed sooner than later. Take the work done in the master branch and port it to the 3.5 branch ttl_nodes 9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 6 days ago 0|i3bamf:
ZooKeeper ZOOKEEPER-2718

org.apache.zookeeper.server.quorum.StandaloneDisabledTest fails intermittently

Test Closed Major Fixed Michael Han Sneha Kanekar Sneha Kanekar 10/Mar/17 03:45   17/May/17 23:43 14/Mar/17 12:06 3.6.0 3.5.3, 3.6.0 quorum, server, tests   0 4   Ubuntu:14.04 The test-suite org.apache.zookeeper.server.quorum.StandaloneDisabledTest fails intermittently with a timeout error. It fails on x86 and ppc64le architechture. The standard output is as follows:

{code:borderStyle=solid}

Testsuite: org.apache.zookeeper.server.quorum.StandaloneDisabledTest
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0 sec

Testcase: startSingleServerTest took 0.001 sec
Caused an ERROR
Timeout occurred. Please note the time in the report does not reflect the time until the timeout.
junit.framework.AssertionFailedError: Timeout occurred. Please note the time in the report does not reflect the time until the timeout.

{code}
ppc64le, x86 9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 1 week, 2 days ago 0|i3b8cn:
ZooKeeper ZOOKEEPER-2717

org.apache.zookeeper.server.quorum.RaceConditionTest fails intermittently

Test Open Major Unresolved Unassigned Sneha Kanekar Sneha Kanekar 10/Mar/17 02:17   04/Apr/17 00:50   3.6.0   quorum, server, tests   0 3   Ubuntu:14.04 The test-suite org.apache.zookeeper.server.quorum.RaceConditionTest fails intermittently on ppc64le and x86 architechture with following error message:
{code:borderStyle=solid}
org.apache.zookeeper.server.quorum.RaceConditionTest.testRaceConditionBetweenLeaderAndAckRequestProcessor

Stacktrace:

Leader failed to transition to new state. Current state is leading
junit.framework.AssertionFailedError: Leader failed to transition to new state. Current state is leading
at org.apache.zookeeper.server.quorum.RaceConditionTest.testRaceConditionBetweenLeaderAndAckRequestProcessor(RaceConditionTest.java:82)
at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:79)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.lang.Thread.run(Thread.java:745)

{code}

Also I have attached the standard output log file.
ppc64le, x86 9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
2 years, 50 weeks, 2 days ago 0|i3b867:
ZooKeeper ZOOKEEPER-2716

Flaky Test: org.apache.zookeeper.server.SessionTrackerTest.testAddSessionAfterSessionExpiry

Test Closed Major Fixed Michael Han Michael Han Michael Han 09/Mar/17 16:46   31/Mar/17 05:01 16/Mar/17 06:44 3.4.9, 3.5.2 3.4.10, 3.5.3, 3.6.0 server, tests   0 3   This test fail once in a while because the test logic has time assumptions that should be fixed. PR in a minute. Sample log when it fails:

{noformat}
Error Message

Duplicate session expiry request has been generated expected:<1> but was:<0>
Stacktrace

junit.framework.AssertionFailedError: Duplicate session expiry request has been generated expected:<1> but was:<0>
at org.apache.zookeeper.server.SessionTrackerTest.testAddSessionAfterSessionExpiry(SessionTrackerTest.java:82)
at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:79)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.lang.Thread.run(Thread.java:745)
Standard Output

2017-03-04 19:25:16,141 [myid:] - INFO [main:JUnit4ZKTestRunner@47] - No test.method specified. using default methods.
2017-03-04 19:25:16,358 [myid:] - INFO [main:JUnit4ZKTestRunner@47] - No test.method specified. using default methods.
2017-03-04 19:25:16,419 [myid:] - INFO [main:ZKTestCase$1@58] - STARTING testAddSessionAfterSessionExpiry
2017-03-04 19:25:16,430 [myid:] - INFO [Time-limited test:JUnit4ZKTestRunner$LoggedInvokeMethod@77] - RUNNING TEST METHOD testAddSessionAfterSessionExpiry
2017-03-04 19:25:16,581 [myid:] - INFO [Time-limited test:Environment@109] - Server environment:zookeeper.version=3.5.3-alpha-SNAPSHOT-6d9fc04c052adbc79bbbb1c63f3f00c816fb8e56, built on 03/04/2017 19:24 GMT
2017-03-04 19:25:16,581 [myid:] - INFO [Time-limited test:Environment@109] - Server environment:host.name=jenkins-ubuntu3.apache.org
2017-03-04 19:25:16,581 [myid:] - INFO [Time-limited test:Environment@109] - Server environment:java.version=1.8.0_121
2017-03-04 19:25:16,581 [myid:] - INFO [Time-limited test:Environment@109] - Server environment:java.vendor=Oracle Corporation
2017-03-04 19:25:16,582 [myid:] - INFO [Time-limited test:Environment@109] - Server environment:java.home=/usr/local/asfpackages/java/jdk1.8.0_121/jre
2017-03-04 19:25:16,587 [myid:] - INFO [Time-limited test:Environment@109] - Server environment:java.class.path=/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch35_jdk8/build/test/classes:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch35_jdk8/build/test/lib/antlr-2.7.7.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch35_jdk8/build/test/lib/antlr4-runtime-4.5.1-1.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch35_jdk8/build/test/lib/checkstyle-6.13.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch35_jdk8/build/test/lib/commons-beanutils-1.9.2.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch35_jdk8/build/test/lib/commons-cli-1.3.1.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch35_jdk8/build/test/lib/commons-collections-3.2.2.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch35_jdk8/build/test/lib/commons-lang3-3.4.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch35_jdk8/build/test/lib/commons-logging-1.1.1.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch35_jdk8/build/test/lib/guava-18.0.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch35_jdk8/build/test/lib/hamcrest-core-1.3.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch35_jdk8/build/test/lib/junit-4.12.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch35_jdk8/build/test/lib/mockito-all-1.8.2.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch35_jdk8/build/classes:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch35_jdk8/src/java/lib/ivy-2.4.0.jar:/home/jenkins/tools/ant/latest/lib/ant.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch35_jdk8/build/lib/commons-cli-1.2.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch35_jdk8/build/lib/jackson-core-asl-1.9.11.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch35_jdk8/build/lib/jackson-mapper-asl-1.9.11.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch35_jdk8/build/lib/javacc.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch35_jdk8/build/lib/javax.servlet-api-3.1.0.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch35_jdk8/build/lib/jetty-http-9.2.18.v20160721.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch35_jdk8/build/lib/jetty-io-9.2.18.v20160721.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch35_jdk8/build/lib/jetty-security-9.2.18.v20160721.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch35_jdk8/build/lib/jetty-server-9.2.18.v20160721.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch35_jdk8/build/lib/jetty-servlet-9.2.18.v20160721.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch35_jdk8/build/lib/jetty-util-9.2.18.v20160721.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch35_jdk8/build/lib/jline-2.11.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch35_jdk8/build/lib/log4j-1.2.17.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch35_jdk8/build/lib/netty-3.10.5.Final.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch35_jdk8/build/lib/slf4j-api-1.7.5.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch35_jdk8/build/lib/slf4j-log4j12-1.7.5.jar:/usr/local/asfpackages/ant/apache-ant-1.10.1/lib/ant-launcher.jar:/home/jenkins/tools/ant/latest/lib/ant-junit.jar:/home/jenkins/tools/ant/latest/lib/ant-junit4.jar
2017-03-04 19:25:16,588 [myid:] - INFO [Time-limited test:Environment@109] - Server environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2017-03-04 19:25:16,588 [myid:] - INFO [Time-limited test:Environment@109] - Server environment:java.io.tmpdir=/tmp
2017-03-04 19:25:16,589 [myid:] - INFO [Time-limited test:Environment@109] - Server environment:java.compiler=<NA>
2017-03-04 19:25:16,590 [myid:] - INFO [Time-limited test:Environment@109] - Server environment:os.name=Linux
2017-03-04 19:25:16,590 [myid:] - INFO [Time-limited test:Environment@109] - Server environment:os.arch=amd64
2017-03-04 19:25:16,592 [myid:] - INFO [Time-limited test:Environment@109] - Server environment:os.version=4.4.0-31-generic
2017-03-04 19:25:16,603 [myid:] - INFO [Time-limited test:Environment@109] - Server environment:user.name=jenkins
2017-03-04 19:25:16,604 [myid:] - INFO [Time-limited test:Environment@109] - Server environment:user.home=/home/jenkins
2017-03-04 19:25:16,604 [myid:] - INFO [Time-limited test:Environment@109] - Server environment:user.dir=/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch35_jdk8/build/test
2017-03-04 19:25:16,604 [myid:] - INFO [Time-limited test:Environment@109] - Server environment:os.memory.free=347MB
2017-03-04 19:25:16,604 [myid:] - INFO [Time-limited test:Environment@109] - Server environment:os.memory.max=455MB
2017-03-04 19:25:16,604 [myid:] - INFO [Time-limited test:Environment@109] - Server environment:os.memory.total=362MB
2017-03-04 19:25:16,624 [myid:] - INFO [Time-limited test:ZooKeeperServer@907] - minSessionTimeout set to 6000
2017-03-04 19:25:16,624 [myid:] - INFO [Time-limited test:ZooKeeperServer@916] - maxSessionTimeout set to 60000
2017-03-04 19:25:16,625 [myid:] - INFO [Time-limited test:ZooKeeperServer@159] - Created server with tickTime 3000 minSessionTimeout 6000 maxSessionTimeout 60000 datadir /home/jenkins/jenkins-slave/workspace/ZooKeeper_branch35_jdk8/build/test/tmp/test1620981504756158168.junit.dir/version-2 snapdir /home/jenkins/jenkins-slave/workspace/ZooKeeper_branch35_jdk8/build/test/tmp/test1620981504756158168.junit.dir/version-2
2017-03-04 19:25:20,434 [myid:] - INFO [SessionTracker:ZooKeeperServer@391] - Expiring session 0x52fbc, timeout of 3000ms exceeded
2017-03-04 19:25:20,450 [myid:] - INFO [Time-limited test:JUnit4ZKTestRunner$LoggedInvokeMethod@98] - TEST METHOD FAILED testAddSessionAfterSessionExpiry
java.lang.AssertionError: Duplicate session expiry request has been generated expected:<1> but was:<0>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:834)
at org.junit.Assert.assertEquals(Assert.java:645)
at org.apache.zookeeper.server.SessionTrackerTest.testAddSessionAfterSessionExpiry(SessionTrackerTest.java:82)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:79)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.lang.Thread.run(Thread.java:745)
2017-03-04 19:25:20,457 [myid:] - INFO [main:ZKTestCase$1@73] - FAILED testAddSessionAfterSessionExpiry
java.lang.AssertionError: Duplicate session expiry request has been generated expected:<1> but was:<0>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:834)
at org.junit.Assert.assertEquals(Assert.java:645)
at org.apache.zookeeper.server.SessionTrackerTest.testAddSessionAfterSessionExpiry(SessionTrackerTest.java:82)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:79)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.lang.Thread.run(Thread.java:745)
2017-03-04 19:25:20,457 [myid:] - INFO [main:ZKTestCase$1@63] - FINISHED testAddSessionAfterSessionExpiry
2017-03-04 19:25:20,462 [myid:] - INFO [main:ZKTestCase$1@58] - STARTING testCloseSessionRequestAfterSessionExpiry
2017-03-04 19:25:20,475 [myid:] - INFO [Time-limited test:JUnit4ZKTestRunner$LoggedInvokeMethod@77] - RUNNING TEST METHOD testCloseSessionRequestAfterSessionExpiry
2017-03-04 19:25:20,476 [myid:] - INFO [Time-limited test:ZooKeeperServer@907] - minSessionTimeout set to 6000
2017-03-04 19:25:20,476 [myid:] - INFO [Time-limited test:ZooKeeperServer@916] - maxSessionTimeout set to 60000
2017-03-04 19:25:20,476 [myid:] - INFO [Time-limited test:ZooKeeperServer@159] - Created server with tickTime 3000 minSessionTimeout 6000 maxSessionTimeout 60000 datadir /home/jenkins/jenkins-slave/workspace/ZooKeeper_branch35_jdk8/build/test/tmp/test8565681305854282002.junit.dir/version-2 snapdir /home/jenkins/jenkins-slave/workspace/ZooKeeper_branch35_jdk8/build/test/tmp/test8565681305854282002.junit.dir/version-2
2017-03-04 19:25:26,431 [myid:] - INFO [SessionTracker:ZooKeeperServer@391] - Expiring session 0x52fbc, timeout of 3000ms exceeded
2017-03-04 19:25:26,431 [myid:] - INFO [SessionTracker:ZooKeeperServer@391] - Expiring session 0x52fbc, timeout of 3000ms exceeded
2017-03-04 19:25:26,432 [myid:] - INFO [Time-limited test:JUnit4ZKTestRunner$LoggedInvokeMethod@82] - Memory used 25301
2017-03-04 19:25:26,432 [myid:] - INFO [Time-limited test:JUnit4ZKTestRunner$LoggedInvokeMethod@87] - Number of threads 11
2017-03-04 19:25:26,432 [myid:] - INFO [Time-limited test:JUnit4ZKTestRunner$LoggedInvokeMethod@102] - FINISHED TEST METHOD testCloseSessionRequestAfterSessionExpiry
2017-03-04 19:25:26,432 [myid:] - INFO [main:ZKTestCase$1@68] - SUCCEEDED testCloseSessionRequestAfterSessionExpiry
2017-03-04 19:25:26,432 [myid:] - INFO [main:ZKTestCase$1@63] - FINISHED testCloseSessionRequestAfterSessionExpiry
{noformat}
flaky, flaky-build, flaky-test 9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 1 week ago 0|i3b7cf:
ZooKeeper ZOOKEEPER-2715

Sessions Expire due to Network partitioning in Zookeeper

Bug Open Major Unresolved Unassigned Tharindu Kumara Tharindu Kumara 09/Mar/17 02:12   10/Mar/17 00:38   3.4.9   c client   0 3   Recently, carried out a test to to find the behavior of clients when a zookeeper server is isolated from the zookeeper leader.

Here used a ensemble of 3 zookeeper servers called A, B and C. And quorum was set up like below.

A - Follower
B - Leader
C - Follower​

A <==> B <==> C
I____________I

And 3 clients are connected to ensemble like below.

C1 is connected A. Both C1 and A are in the same machine.
C2 is connected B. Both C2 and B are in the same machine.
C3 is connected C. Both C3 and C are in the same machine.

To remove the network link between B and C iptables utility is used.

command used:
iptables -I INPUT -s Server_B_IP -j DROP
iptables -I INPUT -s Server_C_IP -j DROP

After removing the link connections looks like below.

A <===> B C
I________I

Simply there is no way to send any packets from zookeeper server B to zookeeper server C and vice versa. But the connection exists between between B and C. And also there is no way to send any packets from B to C3 and vice versa. But the connection exists between between B and C3.

Here What I noticed is that the client connected to Zookeeper Server "C", could not connect to the ensemble, resulting a session expiration timeout.

For this experiment I used tickTime of 3000ms and client session expiration timeout of 45000ms. And tested with different combinations also.
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 2 weeks ago 0|i3b5vj:
ZooKeeper ZOOKEEPER-2714

Zookeeper (standalone) failed to start up

Bug Resolved Blocker Cannot Reproduce Unassigned Daniel C Daniel C 08/Mar/17 18:17   09/Mar/17 11:48 09/Mar/17 11:48 3.4.6   server   0 3   LINUX We've a standalone ZK setup. Upon restart, it failed to serve requests.

Here are the logs:
------------------
2017-03-05 17:33:58,888 [myid:] - INFO [main:QuorumPeerConfig@103] - Reading configuration from: /zookeeper/zookeeper-3.4.6/conf/zoo.1.cfg
2017-03-05 17:33:58,898 [myid:] - WARN [main:QuorumPeerConfig@293] - No server failure will be tolerated. You need at least 3 servers.
2017-03-05 17:33:58,898 [myid:] - INFO [main:QuorumPeerConfig@340] - Defaulting to majority quorums
2017-03-05 17:33:58,909 [myid:1] - INFO [main:DatadirCleanupManager@78] - autopurge.snapRetainCount set to 10
2017-03-05 17:33:58,910 [myid:1] - INFO [main:DatadirCleanupManager@79] - autopurge.purgeInterval set to 5
2017-03-05 17:33:58,911 [myid:1] - INFO [PurgeTask:DatadirCleanupManager$PurgeTask@138] - Purge task started.
2017-03-05 17:33:58,946 [myid:1] - INFO [main:QuorumPeerMain@127] - Starting quorum peer
2017-03-05 17:33:58,966 [myid:1] - INFO [PurgeTask:DatadirCleanupManager$PurgeTask@144] - Purge task completed.
2017-03-05 17:33:58,991 [myid:1] - INFO [main:NIOServerCnxnFactory@94] - binding to port 0.0.0.0/0.0.0.0:2181
2017-03-05 17:33:59,016 [myid:1] - INFO [main:QuorumPeer@959] - tickTime set to 2000
2017-03-05 17:33:59,016 [myid:1] - INFO [main:QuorumPeer@979] - minSessionTimeout set to -1
2017-03-05 17:33:59,016 [myid:1] - INFO [main:QuorumPeer@990] - maxSessionTimeout set to -1
2017-03-05 17:33:59,016 [myid:1] - INFO [main:QuorumPeer@1005] - initLimit set to 20
2017-03-05 17:34:01,328 [myid:1] - INFO [main:QuorumPeer@473] - currentEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
2017-03-05 17:34:01,332 [myid:1] - INFO [main:QuorumPeer@488] - acceptedEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
2017-03-05 17:34:01,335 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket connection from /10.245.66.147:48198
2017-03-05 17:34:01,339 [myid:1] - INFO [Thread-4:QuorumCnxManager$Listener@504] - My election bind port: server001-internal/10.245.66.137:3888
2017-03-05 17:34:01,346 [myid:1] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2017-03-05 17:34:01,346 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007] - Closed socket connection for client /10.245.66.147:48198 (no session established for client)
2017-03-05 17:34:01,346 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket connection from /10.245.66.147:48199
2017-03-05 17:34:01,347 [myid:1] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2017-03-05 17:34:01,347 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007] - Closed socket connection for client /10.245.66.147:48199 (no session established for client)
2017-03-05 17:34:01,347 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket connection from /10.245.66.147:48200
2017-03-05 17:34:01,347 [myid:1] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2017-03-05 17:34:01,348 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007] - Closed socket connection for client /10.245.66.147:48200 (no session established for client)
2017-03-05 17:34:01,348 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket connection from /10.245.66.147:48201
2017-03-05 17:34:01,348 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket connection from /10.245.66.137:46628
2017-03-05 17:34:01,348 [myid:1] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
------------------

Is it a race issue during startup?
2017-03-05 17:34:01,346 [myid:1] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running

9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 2 weeks ago 0|i3b5cv:
ZooKeeper ZOOKEEPER-2713

Create CVE text for ZOOKEEPER-2693 "DOS attack on wchp/wchc four letter words (4lw)"

Task Resolved Blocker Information Provided Patrick D. Hunt Patrick D. Hunt Patrick D. Hunt 08/Mar/17 11:21   19/Dec/19 18:01 09/Oct/17 14:04   3.4.11, 3.5.4 security   0 2   We need to agree to the CVE text for ZOOKEEPER-2693. Let's use the comments here to do so.

The assigned CVE number is CVE-2017-5637
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
2 years, 23 weeks, 3 days ago 0|i3b4jr:
ZooKeeper ZOOKEEPER-2712

MiniKdc test case intermittently failing due to principal not found in Kerberos database

Bug Closed Critical Fixed Rakesh Radhakrishnan Rakesh Radhakrishnan Rakesh Radhakrishnan 08/Mar/17 10:04   19/Jul/17 08:56 21/Mar/17 21:52   3.4.10 tests   0 4   MiniKdc test cases are intermittently failing due to not finding the principal. Below is the failure stacktrace.
{code}
2017-03-08 13:21:10,843 [myid:] - ERROR [NioProcessor-1:AuthenticationService@187] - Error while searching for client learner@EXAMPLE.COM : Client not found in Kerberos database
2017-03-08 13:21:10,843 [myid:] - WARN [NioProcessor-2:KerberosProtocolHandler@241] - Server not found in Kerberos database (7)
2017-03-08 13:21:10,845 [myid:] - WARN [NioProcessor-2:KerberosProtocolHandler@242] - Server not found in Kerberos database (7)
2017-03-08 13:21:10,844 [myid:] - WARN [NioProcessor-1:KerberosProtocolHandler@241] - Client not found in Kerberos database (6)
2017-03-08 13:21:10,845 [myid:] - WARN [NioProcessor-1:KerberosProtocolHandler@242] - Client not found in Kerberos database (6)
{code}

Will attach the detailed log to jira.
9223372036854775807 No Perforce job exists for this issue. 2 9223372036854775807
3 years, 1 day ago 0|i3b4ev:
ZooKeeper ZOOKEEPER-2711

Deadlock between concurrent 4LW commands that iterate over connections with Netty server

Bug Open Critical Unresolved Josh Elser Josh Elser Josh Elser 07/Mar/17 13:19   31/Jan/19 04:46           0 5 0 2400   Observed the following issue in some $dayjob testing environments. Line numbers are a little off compared to master/branch-3.5, but I did confirm the same issue exists there.

With the NettyServerCnxnFactory, before a request is dispatched, the code synchronizes on the {{NettyServerCnxn}} object. However, with some 4LW commands (like {{stat}}), each {{ServerCnxn}} object is also synchronized to (safely) iterate over the internal contents of the object to generate the necessary debug message. As such, multiple concurrent {{stat}} commands can both lock their own {{NettyServerCnxn}} objects, and then be blocked waiting to lock each others' {{ServerCnxn}} in the {{StatCommand}}, deadlocked.
{noformat}
"New I/O worker #55":
at org.apache.zookeeper.server.ServerCnxn.dumpConnectionInfo(ServerCnxn.java:407)
- waiting to lock <0x00000000fabc01b8> (a org.apache.zookeeper.server.NettyServerCnxn)
at org.apache.zookeeper.server.NettyServerCnxn$StatCommand.commandRun(NettyServerCnxn.java:478)
at org.apache.zookeeper.server.NettyServerCnxn$CommandThread.run(NettyServerCnxn.java:311)
at org.apache.zookeeper.server.NettyServerCnxn$CommandThread.start(NettyServerCnxn.java:306)
at org.apache.zookeeper.server.NettyServerCnxn.checkFourLetterWord(NettyServerCnxn.java:677)
at org.apache.zookeeper.server.NettyServerCnxn.receiveMessage(NettyServerCnxn.java:790)
at org.apache.zookeeper.server.NettyServerCnxnFactory$CnxnChannelHandler.processMessage(NettyServerCnxnFactory.java:211)
at org.apache.zookeeper.server.NettyServerCnxnFactory$CnxnChannelHandler.messageReceived(NettyServerCnxnFactory.java:135)
- locked <0x00000000fab68178> (a org.apache.zookeeper.server.NettyServerCnxn)
at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:109)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:90)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
"New I/O worker #51":
at org.apache.zookeeper.server.ServerCnxn.dumpConnectionInfo(ServerCnxn.java:407)
- waiting to lock <0x00000000fab68178> (a org.apache.zookeeper.server.NettyServerCnxn)
at org.apache.zookeeper.server.NettyServerCnxn$StatCommand.commandRun(NettyServerCnxn.java:478)
at org.apache.zookeeper.server.NettyServerCnxn$CommandThread.run(NettyServerCnxn.java:311)
at org.apache.zookeeper.server.NettyServerCnxn$CommandThread.start(NettyServerCnxn.java:306)
at org.apache.zookeeper.server.NettyServerCnxn.checkFourLetterWord(NettyServerCnxn.java:677)
at org.apache.zookeeper.server.NettyServerCnxn.receiveMessage(NettyServerCnxn.java:790)
at org.apache.zookeeper.server.NettyServerCnxnFactory$CnxnChannelHandler.processMessage(NettyServerCnxnFactory.java:211)
at org.apache.zookeeper.server.NettyServerCnxnFactory$CnxnChannelHandler.messageReceived(NettyServerCnxnFactory.java:135)
- locked <0x00000000fabc01b8> (a org.apache.zookeeper.server.NettyServerCnxn)
at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:109)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:90)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
{noformat}
It would appear that the synchronization on the {{NettyServerCnxn}} in {{NettyServerCnxnFactory}} is to blame (and I can see why it was done originally). I think we can just use a different Object (and monitor) to provide mutual exclusion at Netty layer (and avoid synchronization issues at the "application" layer).

 
100% 100% 2400 0 pull-request-available 9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
1 year, 17 weeks ago 0|i3b2gv:
ZooKeeper ZOOKEEPER-2710

Regenerate documentation for branch-3.4 release

Bug Closed Major Fixed Rakesh Radhakrishnan Rakesh Radhakrishnan Rakesh Radhakrishnan 07/Mar/17 12:06   31/Mar/17 05:01 07/Mar/17 12:34 3.4.10 3.4.10 documentation   0 2   This jira can be used to regenerate the documentation as some of the recent commits didn't regenerated the doc section. 9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 2 weeks, 2 days ago 0|i3b2bz:
ZooKeeper ZOOKEEPER-2709

Clarify documentation around "auth" ACL scheme

Task Closed Minor Fixed Josh Elser Josh Elser Josh Elser 03/Mar/17 16:28   17/May/17 23:44 08/Mar/17 20:49   3.5.3, 3.6.0 documentation   0 4   We recently found up in HBASE-17717 that we were incorrectly setting an ACL on our "sensitive" znodes after the output of {{getACL}} on these nodes didn't match what was expected.

In referencing the documentation about how the {{auth}} ACL scheme was supposed to work, it was unclear if it was a ZooKeeper bug or an HBase bug. After reading some ZooKeeper code, we found that it was an HBase bug, but it would be nice to clarify the docs around this ACL scheme.
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 1 week, 3 days ago 0|i3axvj:
ZooKeeper ZOOKEEPER-2708

TracelogFile not being created.

Bug Open Minor Unresolved Unassigned Angelo Esquivel Angelo Esquivel 03/Mar/17 14:59   03/Mar/17 14:59   3.4.6       0 2   Windows 10 64bit We are configuring Zookeeper with log4j to create a tracelog file separated from the zookeeper.log.

We have test using the following java properties:

call %JAVA% "-DrequestTraceFile" "-Dzookeeper.log.dir=%ZOO_LOG_DIR%" "-Dzookeeper.root.logger=%ZOO_LOG4J_PROP%" -cp "%CLASSPATH%" %ZOOMAIN% "%ZOOCFG%" %*


Is there a way to set this in a separate file?
If not, can this be included in the zookeeper.log content?

Please let us know if there is a way.
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 2 weeks, 6 days ago 0|i3axnz:
ZooKeeper ZOOKEEPER-2707

ZOOKEEPER-2686 Fix "Unexpected bean exists!" issue in WatcherTests

Sub-task Resolved Major Fixed Michael Han Abraham Fine Abraham Fine 23/Feb/17 16:26   10/Apr/17 13:25 10/Apr/17 13:25 3.4.10, 3.5.3 3.4.11, 3.6.0 tests   0 2   All the WatcherTests occasionally fail with:

{code}
Error Message:
Unexpected bean exists! expected:<0> but was:<1>

Stack Trace:
junit.framework.AssertionFailedError: Unexpected bean exists! expected:<0> but was:<1>
at org.apache.zookeeper.test.ClientBase.verifyUnexpectedBeans(ClientBase.java:498)
at org.apache.zookeeper.test.ClientBase.startServer(ClientBase.java:477)
at org.apache.zookeeper.test.ClientBase.setUp(ClientBase.java:460)
at org.apache.zookeeper.test.WatcherTest.setUp(WatcherTest.java:76)
{code}

Here is an example: https://builds.apache.org/job/ZooKeeper_branch35_openjdk7/422/
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
2 years, 49 weeks, 3 days ago 0|i3ajen:
ZooKeeper ZOOKEEPER-2706

checkstyle broken on branch-3.4

Bug Closed Major Fixed Abraham Fine Abraham Fine Abraham Fine 23/Feb/17 14:54   31/Mar/17 05:01 22/Mar/17 22:14   3.4.10     0 4   While working on ZOOKEEPER-2696, [~rakeshr] and I noticed that checkstyle is failing to execute on branch-3.4 with the following error:

{code}
BUILD FAILED
/Users/abefine/cloudera_code/zookeeper/build.xml:1595: Unable to create a Checker: cannot initialize module PackageHtml - Unable to instantiate PackageHtml
{code}

This should essentially be a backport of ZOOKEEPER-412
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years ago 0|i3aj5r:
ZooKeeper ZOOKEEPER-2705

Container node remains indefinitely after session has long expired!

Bug Open Major Unresolved Unassigned Steve Fitzgerald Steve Fitzgerald 23/Feb/17 07:24   24/Feb/17 11:43   3.5.1   quorum   0 3   5 x RHEL 2.6.32-431.29.2.el6.x86_64 Zookeeper version: 3.5.1-alpha
Curator Framework version: 3.2.0

We have a 5 node cluster. When we register a service instance everything is created within zookeeper successfully, e.g. for a service names "fake-test-service" I can see the following created:

1. /api/enablement/fake-test-service
2. /api/enablement/fake-test-service/bb831396-5c55-4456-a7c0-5950ba294fd5

When I abnormally kill (kill -9) the process that the service is registered from I expect both of the above to get removed by zookeeper when it expires the session. But only /api/enablement/fake-test-service/bb831396-5c55-4456-a7c0-5950ba294fd5 gets removed successfully.

Here is a snippet of the log file:

{noformat}
2017-02-23 05:50:00,977 [myid:5] - TRACE [SessionTracker:SessionTrackerImpl@208][] - Session closing: 0x502dbce4df60000
2017-02-23 05:50:00,977 [myid:5] - INFO [SessionTracker:ZooKeeperServer@384][] - Expiring session 0x502dbce4df60000, timeout of 40000ms exceeded
2017-02-23 05:50:00,977 [myid:5] - INFO [SessionTracker:QuorumZooKeeperServer@132][] - Submitting global closeSession request for session 0x502dbce4df60000
2017-02-23 05:50:00,977 [myid:5] - TRACE [ProcessThread(sid:5 cport:-1)::ZooTrace@90][] - :Psessionid:0x502dbce4df60000 type:closeSession cxid:0x0 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a
2017-02-23 05:50:00,978 [myid:5] - TRACE [ProcessThread(sid:5 cport:-1)::SessionTrackerImpl@208][] - Session closing: 0x502dbce4df60000
2017-02-23 05:50:00,978 [myid:5] - INFO [ProcessThread(sid:5 cport:-1)::PrepRequestProcessor@649][] - Processed session termination for sessionid: 0x502dbce4df60000
2017-02-23 05:50:00,978 [myid:5] - DEBUG [ProcessThread(sid:5 cport:-1)::CommitProcessor@340][] - Processing request:: sessionid:0x502dbce4df60000 type:closeSession cxid:0x0 zxid:0x1d00000003 txntype:-11 reqpath:n/a
2017-02-23 05:50:00,978 [myid:5] - DEBUG [ProcessThread(sid:5 cport:-1)::Leader@1066][] - Proposing:: sessionid:0x502dbce4df60000 type:closeSession cxid:0x0 zxid:0x1d00000003 txntype:-11 reqpath:n/a
2017-02-23 05:50:00,981 [myid:5] - TRACE [SyncThread:5:Leader@787][] - Ack zxid: 0x1d00000003
2017-02-23 05:50:00,981 [myid:5] - TRACE [SyncThread:5:Leader@790][] - outstanding proposal: 0x1d00000003
2017-02-23 05:50:00,981 [myid:5] - TRACE [SyncThread:5:Leader@793][] - outstanding proposals all
2017-02-23 05:50:00,982 [myid:5] - TRACE [LearnerHandler-/10.24.128.164:38716:Leader@787][] - Ack zxid: 0x1d00000003
2017-02-23 05:50:00,982 [myid:5] - TRACE [LearnerHandler-/10.24.128.164:38716:Leader@790][] - outstanding proposal: 0x1d00000003
2017-02-23 05:50:00,982 [myid:5] - TRACE [LearnerHandler-/10.24.128.164:38716:Leader@793][] - outstanding proposals all
2017-02-23 05:50:00,982 [myid:5] - TRACE [LearnerHandler-/10.24.128.161:55588:Leader@787][] - Ack zxid: 0x1d00000003
2017-02-23 05:50:00,982 [myid:5] - TRACE [LearnerHandler-/10.24.128.161:55588:Leader@790][] - outstanding proposal: 0x1d00000003
2017-02-23 05:50:00,982 [myid:5] - TRACE [LearnerHandler-/10.24.128.161:55588:Leader@793][] - outstanding proposals all
2017-02-23 05:50:00,982 [myid:5] - DEBUG [LearnerHandler-/10.24.128.161:55588:CommitProcessor@327][] - Committing request:: sessionid:0x502dbce4df60000 type:closeSession cxid:0x0 zxid:0x1d00000003 txntype:-11 reqpath:n/a
2017-02-23 05:50:00,982 [myid:5] - TRACE [LearnerHandler-/10.24.128.162:47580:Leader@787][] - Ack zxid: 0x1d00000003
2017-02-23 05:50:00,982 [myid:5] - TRACE [LearnerHandler-/10.24.128.162:47580:Leader@793][] - outstanding proposals all
2017-02-23 05:50:00,983 [myid:5] - DEBUG [LearnerHandler-/10.24.128.162:47580:Leader@808][] - outstanding is 0
2017-02-23 05:50:00,983 [myid:5] - TRACE [LearnerHandler-/10.24.128.160:41119:Leader@787][] - Ack zxid: 0x1d00000003
2017-02-23 05:50:00,983 [myid:5] - TRACE [LearnerHandler-/10.24.128.160:41119:Leader@793][] - outstanding proposals all
2017-02-23 05:50:00,983 [myid:5] - DEBUG [LearnerHandler-/10.24.128.160:41119:Leader@808][] - outstanding is 0
2017-02-23 05:50:00,983 [myid:5] - DEBUG [CommitProcWorkThread-1:FinalRequestProcessor@91][] - Processing request:: sessionid:0x502dbce4df60000 type:closeSession cxid:0x0 zxid:0x1d00000003 txntype:-11 reqpath:n/a
2017-02-23 05:50:00,983 [myid:5] - TRACE [CommitProcWorkThread-1:ZooTrace@90][] - :Esessionid:0x502dbce4df60000 type:closeSession cxid:0x0 zxid:0x1d00000003 txntype:-11 reqpath:n/a
2017-02-23 05:50:00,983 [myid:5] - DEBUG [CommitProcWorkThread-1:DataTree@1034][] - Deleting ephemeral node /api/enablement/fake-test-service/bb831396-5c55-4456-a7c0-5950ba294fd5 for session 0x502dbce4df60000
2017-02-23 05:50:00,983 [myid:5] - DEBUG [CommitProcWorkThread-1:SessionTrackerImpl@218][] - Removing session 0x502dbce4df60000
2017-02-23 05:50:00,983 [myid:5] - TRACE [CommitProcWorkThread-1:ZooTrace@71][] - SessionTrackerImpl --- Removing session 0x502dbce4df60000
2017-02-23 05:50:00,984 [myid:5] - DEBUG [CommitProcWorkThread-1:NettyServerCnxnFactory@411][] - closeSession sessionid:0x361092599260774400
2017-02-23 05:50:00,984 [myid:5] - DEBUG [CommitProcWorkThread-1:NettyServerCnxnFactory@411][] - closeSession sessionid:0x361092599260774400
2017-02-23 05:50:03,525 [myid:5] - TRACE [New I/O worker #5:NettyServerCnxnFactory$CnxnChannelHandler@156][] - message received called BigEndianHeapChannelBuffer(ridx=0, widx=12, cap=12)
2017-02-23 05:50:03,527 [myid:5] - DEBUG [New I/O worker #5:NettyServerCnxnFactory$CnxnChannelHandler@160][] - New message [id: 0xd28589b8, /10.24.128.113:41935 => /10.24.128.165:2281] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=12, cap=12) from [id: 0xd28589b8, /10.24.128.113:41935 => /10.24.128.165:2281]
2017-02-23 05:50:03,527 [myid:5] - DEBUG [New I/O worker #5:NettyServerCnxnFactory$CnxnChannelHandler@175][] - 502d2842d930004 queuedBuffer: null
2017-02-23 05:50:03,527 [myid:5] - TRACE [New I/O worker #5:NettyServerCnxnFactory$CnxnChannelHandler@202][] - 502d2842d930004 buf 0x00000008fffffffe0000000b
2017-02-23 05:50:03,527 [myid:5] - DEBUG [New I/O worker #5:NettyServerCnxnFactory$CnxnChannelHandler@221][] - not throttled
2017-02-23 05:50:03,527 [myid:5] - TRACE [New I/O worker #5:NettyServerCnxn@355][] - message readable 12 bblenrem 4
2017-02-23 05:50:03,528 [myid:5] - TRACE [New I/O worker #5:NettyServerCnxn@360][] - 502d2842d930004 bbLen 0x
2017-02-23 05:50:03,528 [myid:5] - TRACE [New I/O worker #5:NettyServerCnxn@375][] - 502d2842d930004 bbLen 0x00000008
2017-02-23 05:50:03,528 [myid:5] - TRACE [New I/O worker #5:NettyServerCnxn@382][] - 502d2842d930004 bbLen len is 8
2017-02-23 05:50:03,528 [myid:5] - TRACE [New I/O worker #5:NettyServerCnxn@302][] - message readable 8 bb len 8 java.nio.HeapByteBuffer[pos=0 lim=8 cap=8]
2017-02-23 05:50:03,529 [myid:5] - TRACE [New I/O worker #5:NettyServerCnxn@306][] - 502d2842d930004 bb 0x
2017-02-23 05:50:03,529 [myid:5] - TRACE [New I/O worker #5:NettyServerCnxn@320][] - after readBytes message readable 0 bb len 0 java.nio.HeapByteBuffer[pos=8 lim=8 cap=8]
2017-02-23 05:50:03,529 [myid:5] - TRACE [New I/O worker #5:NettyServerCnxn@325][] - after readbytes 502d2842d930004 bb 0xfffffffe0000000b
2017-02-23 05:50:03,530 [myid:5] - DEBUG [ProcessThread(sid:5 cport:-1)::SessionTrackerImpl@291][] - Checking session 0x502d2842d930004
2017-02-23 05:50:03,530 [myid:5] - DEBUG [ProcessThread(sid:5 cport:-1)::CommitProcessor@340][] - Processing request:: sessionid:0x502d2842d930004 type:ping cxid:0xfffffffffffffffe zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a
2017-02-23 05:50:03,530 [myid:5] - DEBUG [CommitProcWorkThread-1:FinalRequestProcessor@91][] - Processing request:: sessionid:0x502d2842d930004 type:ping cxid:0xfffffffffffffffe zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a
2017-02-23 05:50:03,530 [myid:5] - DEBUG [CommitProcWorkThread-1:FinalRequestProcessor@178][] - sessionid:0x502d2842d930004 type:ping cxid:0xfffffffffffffffe zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a
2017-02-23 05:50:03,531 [myid:5] - TRACE [New I/O worker #5:NettyServerCnxnFactory$CnxnChannelHandler@267][] - write complete [id: 0xd28589b8, /10.24.128.113:41935 => /10.24.128.165:2281] WRITTEN_AMOUNT: 85
2017-02-23 05:50:04,275 [myid:5] - ERROR [ContainerManagerTask:ContainerManager$1@84][] - Error checking containers
java.lang.NullPointerException
at org.apache.zookeeper.server.ContainerManager.getCandidates(ContainerManager.java:151)
at org.apache.zookeeper.server.ContainerManager.checkContainers(ContainerManager.java:111)
at org.apache.zookeeper.server.ContainerManager$1.run(ContainerManager.java:78)
at java.util.TimerThread.mainLoop(Timer.java:555)
at java.util.TimerThread.run(Timer.java:505)
2017-02-23 05:50:11,569 [myid:5] - TRACE [New I/O worker #2:NettyServerCnxnFactory$CnxnChannelHandler@156][] - message received called BigEndianHeapChannelBuffer(ridx=0, widx=12, cap=12)
2017-02-23 05:50:11,569 [myid:5] - DEBUG [New I/O worker #2:NettyServerCnxnFactory$CnxnChannelHandler@160][] - New message [id: 0x677c2a25, /10.157.130.185:60591 => /10.24.128.165:2181] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=12, cap=12) from [id: 0x677c2a25, /10.157.130.185:60591 => /10.24.128.165:2181]
2017-02-23 05:50:11,570 [myid:5] - DEBUG [New I/O worker #2:NettyServerCnxnFactory$CnxnChannelHandler@175][] - 10145a3f4f803e5 queuedBuffer: null
2017-02-23 05:50:11,570 [myid:5] - TRACE [New I/O worker #2:NettyServerCnxnFactory$CnxnChannelHandler@202][] - 10145a3f4f803e5 buf 0x00000008fffffffe0000000b
{noformat}

I believe the NullPointerException in the log above is what makes it fail to remove the remaining /api/enablement/fake-test-service directory.

Could someone shed some light on why this might be happening?
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 3 weeks, 6 days ago 0|i3aicn:
ZooKeeper ZOOKEEPER-2704

ZOOKEEPER-2703 Run Jepsen against branch-3.5 / master of ZooKeeper

Sub-task Open Major Unresolved Unassigned Michael Han Michael Han 22/Feb/17 20:16   14/Dec/19 06:09   3.5.2 3.7.0 tests   0 5   The [Jepsen report|https://aphyr.com/posts/291-jepsen-zookeeper] on ZooKeeper was using an old version of ZooKeeper (3.4.5). It would be good to run Jepsen on trunk / branch-3.5 and see what happens. This will also give our confidence on the quality of the upcoming 3.5 stable release. test 9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
1 year, 3 weeks, 5 days ago 0|i3ahh3:
ZooKeeper ZOOKEEPER-2703

[MASTER ISSUE] Create benchmark/stability tests

Test Open Major Unresolved Unassigned Jordan Zimmerman Jordan Zimmerman 22/Feb/17 14:47   02/Mar/17 11:07       java client, recipes, tests   0 4   ZOOKEEPER-2704 It would be useful to have objective tests/benchmarks. These tests/benchmarks can be used to validate future changes to ZooKeeper, compare against other similar products (etcd/consul, etc.) or to help promote ZooKeeper. Possible candidates include:

* leader election tests/benchmarks
* service discovery tests/benchmarks
* distributed locks tests/benchmarks
* ...

Note: each test/benchmark should be a sub-task under this master task
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 3 weeks ago 0|i3agtr:
ZooKeeper ZOOKEEPER-2702

zookeeper ensemble took 20 minutes to come back up after leader failed

Bug Open Major Unresolved Unassigned gopalakrishna gopalakrishna 22/Feb/17 07:42   22/Feb/17 07:52   3.4.9       1 4   OS version is ubuntu 14.04(trusty) Zookeeper version : 3.4.9

OS version is ubuntu 14.04(trusty)

Default configuration of zoo.cfg
tickTime=2000
initLimit=10
syncLimit=5

I have setup the zookeeper ensemble with three servers zk1.com, zk2.com, zk3.com.

Initial State:

ZK1(FOLLOWER)---ZK2(LEADER)-------ZK3(FOLLOWER)


This morning, ZK2(LEADER) went down and it became a FOLLOWER with in fraction of seconds. It took 20 minutes for new LEADER to be decided for the ensemble. ZK3 was the new LEADER.

New State:
ZK(FOLLOWER)----ZK2(FOLLOWER)-----ZK3(LEADER) (after 20 minutes).


Can somone help me to debug what happened?

Zookeeper is managing the solr cloud 2shards, 4 nodes.
9223372036854775807 No Perforce job exists for this issue. 3 9223372036854775807
3 years, 4 weeks, 1 day ago 0|i3afyf:
ZooKeeper ZOOKEEPER-2701

Timeout for RecvWorker is too long

Bug Open Major Unresolved Unassigned Jiafu Jiang Jiafu Jiang 20/Feb/17 01:22   16/Jul/18 03:56   3.4.8, 3.4.9, 3.4.10, 3.4.11       2 6   Centos6.5
ZooKeeper 3.4.8
Environment:
I deploy ZooKeeper in a cluster of three nodes. Each node has three network interfaces(eth0, eth1, eth2).

Hostname is used instead of IP address in zoo.cfg, and quorumListenOnAllIPs=true

Probleam:
I start three ZooKeeper servers( node A, node B, and node C) one by one,
when the leader election finishes, node B is the leader.
Then I shutdown one network interface of node A by command "ifdown eth0". The ZooKeeper server on node A will lost connection to node B and node C. In my test, I will take about 20 minites that the ZooKeepr server of node A realizes the event and try to call the QuorumServer.recreateSocketAddress the resolve the hostname.

I try to read the source code, and I find the code in
{code:java|title=QuorumCnxManager.java:|borderStyle=solid}
class RecvWorker extends ZooKeeperThread {
Long sid;
Socket sock;
volatile boolean running = true;
final DataInputStream din;
final SendWorker sw;

RecvWorker(Socket sock, DataInputStream din, Long sid, SendWorker sw) {
super("RecvWorker:" + sid);
this.sid = sid;
this.sock = sock;
this.sw = sw;
this.din = din;
try {
// OK to wait until socket disconnects while reading.
sock.setSoTimeout(0);
} catch (IOException e) {
LOG.error("Error while accessing socket for " + sid, e);
closeSocket(sock);
running = false;
}
}
...
}
{code}
I notice that the soTime is set to 0 in RecvWorker constructor. I think this is reasonable when the IP address of a ZooKeeper server never change, but considering that the IP address of each ZooKeeper server may change, maybe we should better set a timeout here.

I think this is a problem.
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
1 year, 35 weeks, 3 days ago 0|i3ab9r:
ZooKeeper ZOOKEEPER-2700

Force ZooKeeper to generate snapshot

Improvement Open Minor Unresolved Unassigned Flier Lu Flier Lu 17/Feb/17 08:55   30/Jan/19 08:22           0 3 0 600   When cold backup or remote offline sync Zookeeper instances, we need the latest snapshot.

Add a four letter `snap` command to force Zookeeper to generate snapshot.

100% 100% 600 0 pull-request-available 9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 2 weeks, 1 day ago 0|i3a8of:
ZooKeeper ZOOKEEPER-2699

Restrict 4lw commands based on client IP

Bug Resolved Major Won't Fix Mohammad Arshad Mohammad Arshad Mohammad Arshad 17/Feb/17 02:54   28/Apr/17 06:49 22/Feb/17 10:11     security, server   0 3   Currently 4lw commands are executed without authentication and can be accessed from any IP which has access to ZooKeeper server. ZOOKEEPER-2693 attempts to limit the 4lw commands which are enabled by default or enabled by configuration.

In addition to ZOOKEEPER-2693 we should also restrict 4lw commands based on client IP as well. It is required for following scenario
# User wants to enable all the 4lw commands
# User wants to limit the access of the commands which are considered to be safe by default.

*Implementation:*
we can introduce new property 4lw.commands.host.whitelist
# By default we allow all the hosts, but off course only on the 4lw exposed commands as per the ZOOKEEPER-2693
# It can be configured to allow individual IPs(192.168.1.2,192.168.1.3 etc.)
# It can also be configured to allow group of IPs like 192.168.1.*
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
2 years, 46 weeks, 6 days ago 0|i3a807:
ZooKeeper ZOOKEEPER-2698

SSL support for server to server communication

New Feature Resolved Major Duplicate Abraham Fine Abraham Fine Abraham Fine 16/Feb/17 15:28   02/Mar/17 13:36 02/Mar/17 13:36         0 2   9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 5 weeks ago 0|i3a73j:
ZooKeeper ZOOKEEPER-2697

Handle graceful stop of ZookKeeper client

Improvement Resolved Critical Fixed Enrico Olivelli Enrico Olivelli Enrico Olivelli 16/Feb/17 08:18   12/Jul/17 23:08 01/May/17 11:19 3.4.9 3.5.4, 3.6.0 java client   0 4   As seen in ZOOKEEPER-1394 I would like to have the "close" which waits for all background activities to finish.
In tests the method "testableWaitForShutdown" is used.

We can add a new ZooKeeper.close(int tineout) method which will act as testableWaitForShutdown, joining all support threads.
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
2 years, 46 weeks, 3 days ago 0|i3a6fj:
ZooKeeper ZOOKEEPER-2696

Eclipse ant task no longer determines correct classpath for tests after ZOOKEEPER-2689

Bug Closed Major Fixed Abraham Fine Abraham Fine Abraham Fine 14/Feb/17 19:01   31/Mar/17 05:01 27/Feb/17 01:13 3.4.10 3.4.10 build   0 4   Following the changes made in ZOOKEEPER-2689 IDE's using the .classpath file generated by the eclipse ant task (I tested both idea and eclipse) cannot compile the tests. 9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 3 weeks, 3 days ago 0|i3a38v:
ZooKeeper ZOOKEEPER-2695

Handle unknown error for rolling upgrade old client new server scenario

Bug Open Major Unresolved Mohammad Arshad Mohammad Arshad Mohammad Arshad 14/Feb/17 08:38   22/Feb/17 10:04       java client   0 2   In Zookeeper rolling upgrade scenario where server is new but client is old, when sever sends error code which is not understood by the client, client throws NullPointerException.
KeeperException.SystemErrorException should be thrown for all unknown error code.

9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
3 years, 4 weeks, 1 day ago 0|i3a25r:
ZooKeeper ZOOKEEPER-2694

sync CLI command does not wait for result from server

Bug Closed Major Fixed maoling Mohammad Arshad Mohammad Arshad 14/Feb/17 08:12   16/Oct/19 14:59 23/May/19 12:43 3.5.0 3.6.0, 3.5.6 java client   0 3 0 24000   sync CLI command does not wait for result from server. It returns immediately after invoking the sync's asynchronous API.
Executing bellow command does not give the expected result
{{<zkServer>/bin/zkCli.sh -server host:port sync /}}
100% 100% 24000 0 pull-request-available 9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
43 weeks ago 0|i3a247:
ZooKeeper ZOOKEEPER-2693

DOS attack on wchp/wchc four letter words (4lw)

Bug Closed Blocker Fixed Michael Han Patrick D. Hunt Patrick D. Hunt 13/Feb/17 22:33   06/Aug/18 07:58 07/Mar/17 01:26 3.4.0, 3.5.1, 3.5.2 3.4.10, 3.5.3, 3.6.0 security, server   0 8 0 600   The wchp/wchc four letter words can be exploited in a DOS attack on the ZK client port - typically 2181. The following POC attack was recently published on the web:

https://vulners.com/exploitdb/EDB-ID:41277

The most straightforward way to block this attack is to not allow access to the client port to non-trusted clients - i.e. firewall the ZooKeeper service and only allow access to trusted applications using it for coordination.
100% 100% 600 0 pull-request-available 9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
3 years, 1 week ago 0|i3a187:
ZooKeeper ZOOKEEPER-2692

ZOOKEEPER-2686 Fix race condition in testWatchAutoResetWithPending

Sub-task Closed Major Fixed Abraham Fine Abraham Fine Abraham Fine 10/Feb/17 12:53   15/Nov/17 22:16 21/Feb/17 16:40 3.4.9, 3.5.3, 3.6.0 3.4.10, 3.5.3, 3.6.0 tests   0 5   9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
2 years, 18 weeks ago 0|i39x6n:
ZooKeeper ZOOKEEPER-2691

recreateSocketAddresses may recreate the unreachable IP address

Bug Resolved Minor Fixed Jiafu Jiang Jiafu Jiang Jiafu Jiang 10/Feb/17 06:08   20/Jun/17 12:11 26/May/17 18:29 3.4.8, 3.4.9, 3.4.10, 3.5.0, 3.5.1, 3.5.2, 3.4.11 3.4.11     0 5   Centos6.5
Java8
ZooKeeper3.4.8
The QuorumPeer$QuorumServer.recreateSocketAddress() is used to resolved the hostname to a new IP address(InetAddress) when any exception happens to the socket. It will be very useful when a hostname can be resolved to more than one IP address.
But the problem is Java API InetAddress.getByName(String hostname) will always return the first IP address when the hostname can be resolved to more than one IP address, and the first IP address may be unreachable forever. For example, if a machine has two network interfaces: eth0, eth1, say eth0 has ip1, eth1 has ip2, the relationship between hostname and the IP addresses is set in /etc/hosts. When I "close" the eth0 by command "ifdown eth0", the InetAddress.getByName(String hostname) will still return ip1, which is unreachable forever.

So I think it will be better to check the IP address by InetAddress.isReachable(long) and choose the reachable IP address.


I have modified the ZooKeeper source code, and test the new code in my own environment, and it can work very well when I turn down some network interfaces using "ifdown" command.

The original code is:
{code:title=QuorumPeer.java|borderStyle=solid}
public void recreateSocketAddresses() {
InetAddress address = null;
try {
address = InetAddress.getByName(this.hostname);
LOG.info("Resolved hostname: {} to address: {}", this.hostname, address);
this.addr = new InetSocketAddress(address, this.port);
if (this.electionPort > 0){
this.electionAddr = new InetSocketAddress(address, this.electionPort);
}
} catch (UnknownHostException ex) {
LOG.warn("Failed to resolve address: {}", this.hostname, ex);
// Have we succeeded in the past?
if (this.addr != null) {
// Yes, previously the lookup succeeded. Leave things as they are
return;
}
// The hostname has never resolved. Create our InetSocketAddress(es) as unresolved
this.addr = InetSocketAddress.createUnresolved(this.hostname, this.port);
if (this.electionPort > 0){
this.electionAddr = InetSocketAddress.createUnresolved(this.hostname,
this.electionPort);
}
}
}
{code}

After my modification:
{code:title=QuorumPeer.java|borderStyle=solid}
public void recreateSocketAddresses() {
InetAddress address = null;
try {
address = getReachableAddress(this.hostname);
LOG.info("Resolved hostname: {} to address: {}", this.hostname, address);
this.addr = new InetSocketAddress(address, this.port);
if (this.electionPort > 0){
this.electionAddr = new InetSocketAddress(address, this.electionPort);
}
} catch (UnknownHostException ex) {
LOG.warn("Failed to resolve address: {}", this.hostname, ex);
// Have we succeeded in the past?
if (this.addr != null) {
// Yes, previously the lookup succeeded. Leave things as they are
return;
}
// The hostname has never resolved. Create our InetSocketAddress(es) as unresolved
this.addr = InetSocketAddress.createUnresolved(this.hostname, this.port);
if (this.electionPort > 0){
this.electionAddr = InetSocketAddress.createUnresolved(this.hostname,
this.electionPort);
}
}
}

public InetAddress getReachableAddress(String hostname) throws UnknownHostException {
InetAddress[] addresses = InetAddress.getAllByName(hostname);
for (InetAddress a : addresses) {
try {
if (a.isReachable(5000)) {
return a;
}
} catch (IOException e) {
LOG.warn("IP address {} is unreachable", a);
}
}
// All the IP address is unreachable, just return the first one.
return addresses[0];
}
{code}
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
2 years, 39 weeks, 2 days ago 0|i39wjz:
ZooKeeper ZOOKEEPER-2690

Update documentation source for ZOOKEEPER-2574

Bug Resolved Minor Fixed Mark Fenes Michael Han Michael Han 09/Feb/17 14:34   17/Nov/17 05:59 15/Nov/17 15:29 3.4.9, 3.5.2 3.5.4, 3.6.0, 3.4.12 documentation   0 5   In ZOOKEEPER-2574, the documentation change (https://github.com/apache/zookeeper/pull/111/) was done directly on the generated document files instead of on the document source. This JIRA is created to track the work of porting the doc change on the doc source so the change of the doc will not get lost between releases. newbie 9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
2 years, 17 weeks, 6 days ago
Reviewed
0|i39v7r:
ZooKeeper ZOOKEEPER-2689

Fix Kerberos Authentication related test cases

Bug Closed Critical Fixed Rakesh Radhakrishnan Mohammad Arshad Mohammad Arshad 09/Feb/17 10:47   19/Jul/17 08:55 13/Feb/17 13:13 3.4.9 3.4.10 tests   0 4   Following test classes failed when branch-3.4 is run on java 6.
{noformat}
org.apache.zookeeper.server.quorum.auth.MiniKdcTest
org.apache.zookeeper.server.quorum.auth.QuorumKerberosAuthTest
org.apache.zookeeper.server.quorum.auth.QuorumKerberosHostBasedAuthTest
{noformat}

Error message is {{org/apache/kerby/kerberos/kerb/KrbException : Unsupported major.minor version 51.0}}
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 5 weeks, 2 days ago 0|i39urr:
ZooKeeper ZOOKEEPER-2688

rmr leads to "Node does not exist"

Bug Open Major Unresolved Unassigned Gregory Reshetniak Gregory Reshetniak 08/Feb/17 09:23   09/Feb/17 09:03   3.4.9       0 3   Issuing rmr /vault leads to Node does not exist: /vault/core/_lock/_c_e393e8a4d2c984178373be528a25404a-lock-0000000028

I know that rmr is getting deprecated in next version, but I think this might be cluster consistency bug.

Please advice.
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 6 weeks ago 0|i39slr:
ZooKeeper ZOOKEEPER-2687

Deadlock while shutting down the Leader server.

Bug Closed Major Fixed Mohammad Arshad Mohammad Arshad Mohammad Arshad 08/Feb/17 02:19   17/May/17 23:43 15/Feb/17 11:24 3.5.2, 3.6.0 3.5.3, 3.6.0 server   0 7   Leader server enters into deadlock while shutting down. This happens some time only.
The reason and deadlock flow is same as ZOOKEEPER-2380.
shutdown was removed from synchronized block in ZOOKEEPER-2380
Now shutdown is called from synchronized block from another place.
{code}
// check leader running status
if (!this.isRunning()) {
shutdown("Unexpected internal error");
return;
}
{code}
9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
3 years, 5 weeks, 1 day ago 0|i39rz3:
ZooKeeper ZOOKEEPER-2686

Flaky Test: org.apache.zookeeper.test.WatcherTest.

Test Resolved Major Fixed Michael Han Michael Han Michael Han 07/Feb/17 17:01   10/Apr/17 13:25 10/Apr/17 13:25 3.4.9, 3.5.2, 3.6.0 3.4.11, 3.6.0 tests   0 1   ZOOKEEPER-2692, ZOOKEEPER-2707 Once in a while, these tests failed.

{noformat}
org.apache.zookeeper.test.WatcherTest.testWatchAutoResetWithPending
org.apache.zookeeper.test.WatcherTest.testWatcherCorrectness
org.apache.zookeeper.test.WatcherTest.testWatcherAutoResetDisabledWithLocal
org.apache.zookeeper.test.WatcherTest.testWatcherAutoResetWithGlobal
org.apache.zookeeper.test.WatcherTest.testWatcherCount
org.apache.zookeeper.test.WatcherTest.testWatcherAutoResetDisabledWithGlobal
org.apache.zookeeper.test.WatcherTest.testWatcherAutoResetWithLocal
{noformat}
flaky-test 9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 6 weeks, 2 days ago 0|i39r4n:
ZooKeeper ZOOKEEPER-2685

How to implement SSL in zookeeper-3.4.5-3

Bug Resolved Major Won't Fix Unassigned shamim khan shamim khan 07/Feb/17 04:08   07/Feb/17 23:14 07/Feb/17 23:14 3.4.5   java client   0 2   want to implement SSL in zookeeeper. But not able to implement as version issue. So how can we implement in zookeeper-3.4.5-3. 9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 6 weeks, 1 day ago 0|i39pkf:
ZooKeeper ZOOKEEPER-2684

Fix a crashing bug in the mixed workloads commit processor

Bug Resolved Blocker Fixed Kfir Lev-Ari Ryan Zhang Ryan Zhang 07/Feb/17 00:23   30/Jan/19 13:46 03/Nov/17 00:20 3.6.0 3.6.0 server   0 9 0 1200   with pretty heavy load on a real cluster We deployed our build with ZOOKEEPER-2024 and it quickly started to crash with the following error

atla-buh-05-sr1.prod.twttr.net: 2017-01-18 22:24:42,305 - ERROR [CommitProcessor:2] -org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:268) – Got cxid 0x119fa expected 0x11fc5 for client session id 1009079ba470055
atla-buh-05-sr1.prod.twttr.net: 2017-01-18 22:32:04,746 - ERROR [CommitProcessor:2] -org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:268) – Got cxid 0x698 expected 0x928 for client session id 4002eeb3fd0009d
atla-buh-05-sr1.prod.twttr.net: 2017-01-18 22:34:46,648 - ERROR [CommitProcessor:2] -org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:268) – Got cxid 0x8904 expected 0x8f34 for client session id 51b8905c90251
atla-buh-05-sr1.prod.twttr.net: 2017-01-18 22:43:46,834 - ERROR [CommitProcessor:2] -org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:268) – Got cxid 0x3a8d expected 0x3ebc for client session id 2051af11af900cc

clearly something is not right in the new commit processor per session queue implementation.
100% 100% 1200 0 pull-request-available 9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
2 years, 19 weeks, 6 days ago 0|i39pbb:
ZooKeeper ZOOKEEPER-2683

RaceConditionTest is flaky

Bug Closed Major Fixed Mohammad Arshad Mohammad Arshad Mohammad Arshad 06/Feb/17 13:01   17/May/17 23:43 13/Feb/17 16:10 3.5.2, 3.6.0 3.5.3, 3.6.0 tests   0 5   *Error Message*
{noformat}
Leader failed to transition to LOOKING or FOLLOWING state
{noformat}
*Stacktrace*
{noformat}
junit.framework.AssertionFailedError: Leader failed to transition to LOOKING or FOLLOWING state
at org.apache.zookeeper.server.quorum.RaceConditionTest.testRaceConditionBetweenLeaderAndAckRequestProcessor(RaceConditionTest.java:74)
at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:79)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.lang.Thread.run(Thread.java:745)
{noformat}
[CI Failures Reference|https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/279//testReport/org.apache.zookeeper.server.quorum/RaceConditionTest/testRaceConditionBetweenLeaderAndAckRequestProcessor/]
9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
3 years, 5 weeks, 3 days ago 0|i39obj:
ZooKeeper ZOOKEEPER-2682

Make it optional to fail build on test failure

Improvement Closed Minor Fixed Mohammad Arshad Mohammad Arshad Mohammad Arshad 05/Feb/17 06:27   31/Mar/17 05:01 06/Feb/17 21:36   3.4.10, 3.5.3, 3.6.0 build, tests   0 4   Currently if there is a test failure, build is marked as failed and exits.
I want to rerun the failed test cases instead of exiting.
9223372036854775807 No Perforce job exists for this issue. 2 9223372036854775807
3 years, 6 weeks, 2 days ago
Reviewed
0|i39mtb:
ZooKeeper ZOOKEEPER-2681

ConnectionState does not sync startup of ExhibitorEnsembleProvider and Zookeeper connection

Bug Resolved Major Invalid Unassigned Egor Ryashin Egor Ryashin 27/Jan/17 17:02   28/Jan/17 08:15 28/Jan/17 08:15 3.4.5   java client   0 1   Use CuratorFrameworkFactory.Builder and specify ExhibitorEnsembleProvider.
Call build() and start().
Internal ConnectionState.start() calls ensembleProvider.start() which should poll for hostnames to produce connectionString.
Without waiting (for connectionString) ConnectionState calls zooKeeper.closeAndReset() and ClientCnxn is created with empty connectionString. That leads to lame zooKeeper sending requests to localhost.
{noformat}
2017-01-27T22:56:17,618 INFO [Agents-0] org.apache.curator.framework.imps.CuratorFrameworkImpl - Starting
2017-01-27T22:56:17,619 INFO [Agents-0] org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString= sessionTimeout=60001 watcher=org.apache.curator.ConnectionState@4402fad2
2017-01-27T22:56:17,625 INFO [Agents-0-SendThread(127.0.0.1:2181)] org.apache.zookeeper.ClientCnxn - Opening socket connection to server 127.0.0.1/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
2017-01-27T22:56:18,632 WARN [Agents-0-SendThread(127.0.0.1:2181)] org.apache.zookeeper.ClientCnxn - Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused: no further information
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:1.8.0_74]
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) ~[?:1.8.0_74]
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350) ~[zookeeper-3.4.5.jar:3.4.5-1392090]
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068) [zookeeper-3.4.5.jar:3.4.5-1392090]
2017-01-27T22:56:19,733 INFO [Agents-0-SendThread(127.0.0.1:2181)] org.apache.zookeeper.ClientCnxn - Opening socket connection to server 127.0.0.1/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
2017-01-27T22:56:19,807 INFO [Curator-ExhibitorEnsembleProvider-0] org.apache.curator.ensemble.exhibitor.ExhibitorEnsembleProvider - Connection string has changed. Old value (), new value (172.19.2.158:2181,172.19.2.15:2181,172.19.2.177:2181,172.19.2.4:2181,172.19.2.89:2181,172.19.2.72:2181)
2017-01-27T22:56:20,734 WARN [Agents-0-SendThread(127.0.0.1:2181)] org.apache.zookeeper.ClientCnxn - Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused: no further information
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:1.8.0_74]
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) ~[?:1.8.0_74]
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350) ~[zookeeper-3.4.5.jar:3.4.5-1392090]
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068) [zookeeper-3.4.5.jar:3.4.5-1392090]
2017-01-27T22:56:21,835 INFO [Agents-0-SendThread(127.0.0.1:2181)] org.apache.zookeeper.ClientCnxn - Opening socket connection to server 127.0.0.1/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
{noformat}
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 7 weeks, 5 days ago 0|i39bhr:
ZooKeeper ZOOKEEPER-2680

Correct DataNode.getChildren() inconsistent behaviour.

Bug Closed Major Fixed Mohammad Arshad Mohammad Arshad Mohammad Arshad 27/Jan/17 01:03   31/Mar/17 05:01 10/Feb/17 12:29 3.4.9, 3.5.1 3.4.10, 3.5.3, 3.6.0 server   1 7   DataNode.getChildren() API returns null and empty set if there are no children in it depending on when the API is called. DataNode.getChildren() API behavior should be changed and it should always return empty set if the node does not have any child

*DataNode.getChildren() API Current Behavior:*
# returns null initially
When DataNode is created and no children are added yet, DataNode.getChildren() returns null
# returns empty set after all the children are deleted:
created a Node
add a child
delete the child
DataNode.getChildren() returns empty set.

After fix DataNode.getChildren() should return empty set in all the above cases.
9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
3 years, 5 weeks, 3 days ago 0|i39acv:
ZooKeeper ZOOKEEPER-2679

ZOOKEEPER-3170 Flaky Test: org.apache.zookeeper.server.quorum.auth.QuorumAuthUpgradeTest.testRollingUpgrade

Sub-task Resolved Major Cannot Reproduce Andor Molnar Michael Han Michael Han 26/Jan/17 16:57   25/Oct/18 11:01 25/Oct/18 11:01 3.4.10   quorum, security, server, tests   1 4   This flaky test was introduced as part of ZOOKEEPER-1045, which fails regularly both in our internal test bot and upstream apache build bot.

{noformat}
Error Message

waiting for server1being up
Stacktrace

junit.framework.AssertionFailedError: waiting for server1being up
at org.apache.zookeeper.server.quorum.auth.QuorumAuthUpgradeTest.restartServer(QuorumAuthUpgradeTest.java:232)
at org.apache.zookeeper.server.quorum.auth.QuorumAuthUpgradeTest.testRollingUpgrade(QuorumAuthUpgradeTest.java:204)
at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:52)
Standard Output

2017-01-24 22:12:19,965 [myid:] - INFO [main:ZKTestCase$1@50] - STARTING testAuthLearnerAgainstNoAuthRequiredServer
2017-01-24 22:12:19,971 [myid:] - INFO [Thread-0:JUnit4ZKTestRunner$LoggedInvokeMethod@50] - RUNNING TEST METHOD testAuthLearnerAgainstNoAuthRequiredServer
2017-01-24 22:12:19,972 [myid:] - INFO [Thread-0:PortAssignment@32] - assigning port 11221
2017-01-24 22:12:19,973 [myid:] - INFO [Thread-0:PortAssignment@32] - assigning port 11222
2017-01-24 22:12:19,973 [myid:] - INFO [Thread-0:PortAssignment@32] - assigning port 11223
2017-01-24 22:12:19,973 [myid:] - INFO [Thread-0:PortAssignment@32] - assigning port 11224
2017-01-24 22:12:19,973 [myid:] - INFO [Thread-0:PortAssignment@32] - assigning port 11225
2017-01-24 22:12:19,973 [myid:] - INFO [Thread-0:PortAssignment@32] - assigning port 11226
2017-01-24 22:12:19,974 [myid:] - INFO [Thread-0:QuorumPeerTestBase$MainThread@81] - id = 0 tmpDir = /data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test8717551953335097579.junit.dir clientPort = 11221
2017-01-24 22:12:19,981 [myid:] - INFO [Thread-0:QuorumPeerTestBase$MainThread@81] - id = 1 tmpDir = /data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test4158500655544466289.junit.dir clientPort = 11224
2017-01-24 22:12:20,048 [myid:] - INFO [Thread-0:FourLetterWordMain@43] - connecting to 127.0.0.1 11221
2017-01-24 22:12:20,048 [myid:] - INFO [Thread-2:QuorumPeerConfig@111] - Reading configuration from: /data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test4158500655544466289.junit.dir/zoo.cfg
2017-01-24 22:12:20,050 [myid:] - INFO [Thread-1:QuorumPeerConfig@111] - Reading configuration from: /data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test8717551953335097579.junit.dir/zoo.cfg
2017-01-24 22:12:20,054 [myid:] - WARN [Thread-2:QuorumPeerConfig@327] - No server failure will be tolerated. You need at least 3 servers.
2017-01-24 22:12:20,056 [myid:] - INFO [Thread-2:QuorumPeerConfig@374] - Defaulting to majority quorums
2017-01-24 22:12:20,059 [myid:] - WARN [Thread-1:QuorumPeerConfig@327] - No server failure will be tolerated. You need at least 3 servers.
2017-01-24 22:12:20,064 [myid:] - INFO [Thread-0:ClientBase@246] - server 127.0.0.1:11221 not up java.net.ConnectException: Connection refused
2017-01-24 22:12:20,070 [myid:] - INFO [Thread-1:QuorumPeerConfig@374] - Defaulting to majority quorums
2017-01-24 22:12:20,071 [myid:1] - INFO [Thread-2:DatadirCleanupManager@78] - autopurge.snapRetainCount set to 3
2017-01-24 22:12:20,071 [myid:1] - INFO [Thread-2:DatadirCleanupManager@79] - autopurge.purgeInterval set to 0
2017-01-24 22:12:20,071 [myid:1] - INFO [Thread-2:DatadirCleanupManager@101] - Purge task is not scheduled.
2017-01-24 22:12:20,075 [myid:0] - INFO [Thread-1:DatadirCleanupManager@78] - autopurge.snapRetainCount set to 3
2017-01-24 22:12:20,075 [myid:0] - INFO [Thread-1:DatadirCleanupManager@79] - autopurge.purgeInterval set to 0
2017-01-24 22:12:20,075 [myid:0] - INFO [Thread-1:DatadirCleanupManager@101] - Purge task is not scheduled.
2017-01-24 22:12:20,210 [myid:1] - WARN [Thread-2:QuorumPeerMain@129] - Unable to register log4j JMX control
javax.management.InstanceAlreadyExistsException: log4j:hiearchy=default
at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:437)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1898)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:966)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324)
at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
at org.apache.zookeeper.jmx.ManagedUtil.registerLog4jMBeans(ManagedUtil.java:53)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:127)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:116)
at org.apache.zookeeper.server.quorum.QuorumPeerTestBase$MainThread.run(QuorumPeerTestBase.java:140)
at java.lang.Thread.run(Thread.java:745)
2017-01-24 22:12:20,213 [myid:1] - INFO [Thread-2:QuorumPeerMain@132] - Starting quorum peer
2017-01-24 22:12:20,222 [myid:0] - INFO [Thread-1:QuorumPeerMain@132] - Starting quorum peer
2017-01-24 22:12:20,237 [myid:0] - INFO [Thread-1:NIOServerCnxnFactory@94] - binding to port 0.0.0.0/0.0.0.0:11221
2017-01-24 22:12:20,241 [myid:1] - INFO [Thread-2:NIOServerCnxnFactory@94] - binding to port 0.0.0.0/0.0.0.0:11224
2017-01-24 22:12:20,275 [myid:0] - INFO [Thread-1:QuorumPeer@1048] - minSessionTimeout set to -1
2017-01-24 22:12:20,275 [myid:1] - INFO [Thread-2:QuorumPeer@1048] - minSessionTimeout set to -1
2017-01-24 22:12:20,275 [myid:0] - INFO [Thread-1:QuorumPeer@1059] - maxSessionTimeout set to -1
2017-01-24 22:12:20,275 [myid:1] - INFO [Thread-2:QuorumPeer@1059] - maxSessionTimeout set to -1
2017-01-24 22:12:20,277 [myid:0] - INFO [Thread-1:QuorumPeer@1279] - quorum.auth.enableSasl set to true
2017-01-24 22:12:20,277 [myid:1] - INFO [Thread-2:QuorumPeer@1279] - quorum.auth.enableSasl set to true
2017-01-24 22:12:20,277 [myid:0] - INFO [Thread-1:QuorumPeer@1264] - quorum.auth.serverRequireSasl set to false
2017-01-24 22:12:20,277 [myid:1] - INFO [Thread-2:QuorumPeer@1264] - quorum.auth.serverRequireSasl set to false
2017-01-24 22:12:20,277 [myid:0] - INFO [Thread-1:QuorumPeer@1270] - quorum.auth.learnerRequireSasl set to false
2017-01-24 22:12:20,278 [myid:1] - INFO [Thread-2:QuorumPeer@1270] - quorum.auth.learnerRequireSasl set to false
2017-01-24 22:12:20,278 [myid:0] - INFO [Thread-1:QuorumPeer@1286] - quorum.auth.kerberos.servicePrincipal set to zkquorum/localhost
2017-01-24 22:12:20,278 [myid:1] - INFO [Thread-2:QuorumPeer@1286] - quorum.auth.kerberos.servicePrincipal set to zkquorum/localhost
2017-01-24 22:12:20,278 [myid:0] - INFO [Thread-1:QuorumPeer@1298] - quorum.auth.server.saslLoginContext set to QuorumServer
2017-01-24 22:12:20,278 [myid:1] - INFO [Thread-2:QuorumPeer@1298] - quorum.auth.server.saslLoginContext set to QuorumServer
2017-01-24 22:12:20,278 [myid:0] - INFO [Thread-1:QuorumPeer@1292] - quorum.auth.learner.saslLoginContext set to QuorumLearner
2017-01-24 22:12:20,278 [myid:1] - INFO [Thread-2:QuorumPeer@1292] - quorum.auth.learner.saslLoginContext set to QuorumLearner
2017-01-24 22:12:20,278 [myid:0] - INFO [Thread-1:QuorumPeer@1306] - quorum.cnxn.threads.size set to 20
2017-01-24 22:12:20,279 [myid:1] - INFO [Thread-2:QuorumPeer@1306] - quorum.cnxn.threads.size set to 20
2017-01-24 22:12:20,288 [myid:0] - INFO [Thread-1:Login@294] - QuorumServer successfully logged in.
2017-01-24 22:12:20,288 [myid:1] - INFO [Thread-2:Login@294] - QuorumServer successfully logged in.
2017-01-24 22:12:20,292 [myid:0] - INFO [Thread-1:Login@294] - QuorumLearner successfully logged in.
2017-01-24 22:12:20,292 [myid:1] - INFO [Thread-2:Login@294] - QuorumLearner successfully logged in.
2017-01-24 22:12:20,299 [myid:0] - INFO [Thread-1:QuorumPeer@540] - currentEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
2017-01-24 22:12:20,299 [myid:1] - INFO [Thread-2:QuorumPeer@540] - currentEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
2017-01-24 22:12:20,302 [myid:0] - INFO [Thread-1:QuorumPeer@555] - acceptedEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
2017-01-24 22:12:20,302 [myid:1] - INFO [Thread-2:QuorumPeer@555] - acceptedEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
2017-01-24 22:12:20,312 [myid:1] - INFO [Thread-4:QuorumCnxManager$Listener@691] - My election bind port: 0.0.0.0/0.0.0.0:11226
2017-01-24 22:12:20,313 [myid:0] - INFO [Thread-5:QuorumCnxManager$Listener@691] - My election bind port: 0.0.0.0/0.0.0.0:11223
2017-01-24 22:12:20,320 [myid:] - INFO [Thread-0:FourLetterWordMain@43] - connecting to 127.0.0.1 11221
2017-01-24 22:12:20,322 [myid:0] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11221:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:58467
2017-01-24 22:12:20,338 [myid:0] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11221:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:58467
2017-01-24 22:12:20,349 [myid:0] - INFO [Thread-6:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:58467 (no session established for client)
2017-01-24 22:12:20,345 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11221:QuorumPeer@781] - LOOKING
2017-01-24 22:12:20,345 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11224:QuorumPeer@781] - LOOKING
2017-01-24 22:12:20,352 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11224:FastLeaderElection@744] - New election. My id = 1, proposed zxid=0x0
2017-01-24 22:12:20,354 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11221:FastLeaderElection@744] - New election. My id = 0, proposed zxid=0x0
2017-01-24 22:12:20,357 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@542] - Notification: 1 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)
2017-01-24 22:12:20,357 [myid:1] - INFO [localhost/127.0.0.1:11226:QuorumCnxManager$Listener@698] - Received connection request /127.0.0.1:48700
2017-01-24 22:12:20,356 [myid:1] - INFO [QuorumConnectionThread-[myid=1]-1:SaslQuorumAuthLearner@79] - Skipping SASL authentication as quorum.auth.learnerRequireSasl=false
2017-01-24 22:12:20,354 [myid:0] - INFO [localhost/127.0.0.1:11223:QuorumCnxManager$Listener@698] - Received connection request /127.0.0.1:51927
2017-01-24 22:12:20,358 [myid:0] - INFO [QuorumConnectionThread-[myid=0]-1:SaslQuorumAuthLearner@79] - Skipping SASL authentication as quorum.auth.learnerRequireSasl=false
2017-01-24 22:12:20,357 [myid:0] - INFO [WorkerReceiver[myid=0]:FastLeaderElection@542] - Notification: 0 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 0 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)
2017-01-24 22:12:20,359 [myid:0] - INFO [QuorumConnectionThread-[myid=0]-1:QuorumCnxManager@331] - Have smaller server identifier, so dropping the connection: (1, 0)
2017-01-24 22:12:20,363 [myid:1] - WARN [QuorumConnectionThread-[myid=1]-2:SaslQuorumAuthServer@135] - Failed to authenticate using SASL
java.io.EOFException
at java.io.DataInputStream.readFully(DataInputStream.java:197)
at java.io.DataInputStream.readLong(DataInputStream.java:416)
at org.apache.jute.BinaryInputArchive.readLong(BinaryInputArchive.java:67)
at org.apache.zookeeper.server.quorum.auth.QuorumAuth.nextPacketIsAuth(QuorumAuth.java:91)
at org.apache.zookeeper.server.quorum.auth.SaslQuorumAuthServer.authenticate(SaslQuorumAuthServer.java:79)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.handleConnection(QuorumCnxManager.java:435)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.receiveConnection(QuorumCnxManager.java:373)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$QuorumConnectionReceiverThread.run(QuorumCnxManager.java:409)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2017-01-24 22:12:20,363 [myid:1] - WARN [QuorumConnectionThread-[myid=1]-2:SaslQuorumAuthServer@136] - Maintaining learner connection despite SASL authentication failure. server addr: /127.0.0.1:48700, quorum.auth.serverRequireSasl: false
2017-01-24 22:12:20,364 [myid:1] - WARN [SendWorker:0:QuorumCnxManager$SendWorker@896] - Interrupted while waiting for message on queue
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1049)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:73)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:884)
2017-01-24 22:12:20,364 [myid:0] - INFO [localhost/127.0.0.1:11223:QuorumCnxManager$Listener@698] - Received connection request /127.0.0.1:51929
2017-01-24 22:12:20,364 [myid:1] - WARN [SendWorker:0:QuorumCnxManager$SendWorker@906] - Send worker leaving thread
2017-01-24 22:12:20,364 [myid:1] - WARN [RecvWorker:0:QuorumCnxManager$RecvWorker@983] - Interrupting SendWorker
2017-01-24 22:12:20,365 [myid:0] - WARN [SendWorker:1:QuorumCnxManager$SendWorker@901] - Exception when using channel: for id 1 my id = 0 error = java.net.SocketException: Broken pipe
2017-01-24 22:12:20,365 [myid:0] - WARN [SendWorker:1:QuorumCnxManager$SendWorker@906] - Send worker leaving thread
2017-01-24 22:12:20,366 [myid:0] - WARN [RecvWorker:1:QuorumCnxManager$RecvWorker@983] - Interrupting SendWorker
2017-01-24 22:12:20,366 [myid:1] - INFO [QuorumConnectionThread-[myid=1]-3:SaslQuorumAuthLearner@79] - Skipping SASL authentication as quorum.auth.learnerRequireSasl=false
2017-01-24 22:12:20,378 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@542] - Notification: 0 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 0 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)
2017-01-24 22:12:20,379 [myid:0] - INFO [WorkerReceiver[myid=0]:FastLeaderElection@542] - Notification: 1 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)
2017-01-24 22:12:20,380 [myid:0] - INFO [WorkerReceiver[myid=0]:FastLeaderElection@542] - Notification: 1 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 0 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)
2017-01-24 22:12:20,380 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@542] - Notification: 1 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 0 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)
2017-01-24 22:12:20,581 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11221:QuorumPeer@849] - FOLLOWING
2017-01-24 22:12:20,581 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11224:QuorumPeer@861] - LEADING
2017-01-24 22:12:20,585 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11224:Leader@62] - TCP NoDelay set to: true
2017-01-24 22:12:20,588 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11221:Learner@85] - TCP NoDelay set to: true
2017-01-24 22:12:20,593 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11221:Environment@100] - Server environment:zookeeper.version=3.4.5--1, built on 01/25/2017 06:01 GMT
2017-01-24 22:12:20,593 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11221:Environment@100] - Server environment:host.name=ec2-unittest-ub12-m3-large-0e11.vpc.cloudera.com
2017-01-24 22:12:20,594 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11221:Environment@100] - Server environment:java.version=1.7.0_75
2017-01-24 22:12:20,594 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11221:Environment@100] - Server environment:java.vendor=Oracle Corporation
2017-01-24 22:12:20,594 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11221:Environment@100] - Server environment:java.home=/data/jenkins/tools/hudson.model.JDK/Java_7/jre
2017-01-24 22:12:20,594 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11221:Environment@100] - Server environment:java.class.path=/data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/classes:/data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/lib/antlr-2.7.6.jar:/data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/lib/checkstyle-5.0.jar:/data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/lib/commons-beanutils-core-1.7.0.jar:/data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/lib/commons-cli-1.0.jar:/data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/lib/commons-collections-3.2.2.jar:/data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/lib/commons-io-2.4.jar:/data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/lib/commons-lang-1.0.jar:/data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/lib/commons-logging-1.0.3.jar:/data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/lib/google-collections-0.9.jar:/data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/lib/junit-4.8.1.jar:/data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/lib/kerb-admin-1.0.0-RC2.jar:/data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/lib/kerb-client-1.0.0-RC2.jar:/data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/lib/kerb-common-1.0.0-RC2.jar:/data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/lib/kerb-core-1.0.0-RC2.jar:/data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/lib/kerb-crypto-1.0.0-RC2.jar:/data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/lib/kerb-identity-1.0.0-RC2.jar:/data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/lib/kerb-server-1.0.0-RC2.jar:/data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/lib/kerb-simplekdc-1.0.0-RC2.jar:/data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/lib/kerb-util-1.0.0-RC2.jar:/data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/lib/kerby-asn1-1.0.0-RC2.jar:/data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/lib/kerby-config-1.0.0-RC2.jar:/data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/lib/kerby-pkix-1.0.0-RC2.jar:/data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/lib/kerby-util-1.0.0-RC2.jar:/data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/lib/log4j-1.2.17.jar:/data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/lib/mockito-all-1.8.2.jar:/data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/lib/slf4j-api-1.7.14.jar:/data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/lib/slf4j-log4j12-1.7.14.jar:/data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/classes:/data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/src/java/lib/ivy-2.2.0.jar:/data/jenkins/tools/hudson.tasks.Ant_AntInstallation/Ant_1.8.2/apache-ant-1.9.6/lib/ant.jar:/data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/lib/jline-2.11.jar:/data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/lib/log4j-1.2.16.jar:/data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/lib/netty-3.10.5.Final.jar:/data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/lib/slf4j-api-1.7.5.jar:/data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/lib/slf4j-log4j12-1.7.5.jar:/mnt/toolchain/clover-ant-4.0.3/lib/clover.jar:/data/jenkins/tools/hudson.tasks.Ant_AntInstallation/Ant_1.8.2/apache-ant-1.9.6/lib/ant-launcher.jar:/data/jenkins/tools/hudson.tasks.Ant_AntInstallation/Ant_1.8.2/apache-ant-1.9.6/lib/ant-junit.jar:/data/jenkins/tools/hudson.tasks.Ant_AntInstallation/Ant_1.8.2/apache-ant-1.9.6/lib/ant-junit4.jar
2017-01-24 22:12:20,594 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11221:Environment@100] - Server environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2017-01-24 22:12:20,594 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11221:Environment@100] - Server environment:java.io.tmpdir=/tmp
2017-01-24 22:12:20,594 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11221:Environment@100] - Server environment:java.compiler=<NA>
2017-01-24 22:12:20,594 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11221:Environment@100] - Server environment:os.name=Linux
2017-01-24 22:12:20,594 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11221:Environment@100] - Server environment:os.arch=amd64
2017-01-24 22:12:20,594 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11221:Environment@100] - Server environment:os.version=3.2.0-33-virtual
2017-01-24 22:12:20,594 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11221:Environment@100] - Server environment:user.name=jenkins
2017-01-24 22:12:20,594 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11221:Environment@100] - Server environment:user.home=/var/lib/jenkins
2017-01-24 22:12:20,595 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11221:Environment@100] - Server environment:user.dir=/data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7
2017-01-24 22:12:20,597 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11224:ZooKeeperServer@162] - Created server with tickTime 4000 minSessionTimeout 8000 maxSessionTimeout 80000 datadir /data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test4158500655544466289.junit.dir/data/version-2 snapdir /data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test4158500655544466289.junit.dir/data/version-2
2017-01-24 22:12:20,597 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11221:ZooKeeperServer@162] - Created server with tickTime 4000 minSessionTimeout 8000 maxSessionTimeout 80000 datadir /data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test8717551953335097579.junit.dir/data/version-2 snapdir /data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test8717551953335097579.junit.dir/data/version-2
2017-01-24 22:12:20,599 [myid:] - INFO [Thread-0:FourLetterWordMain@43] - connecting to 127.0.0.1 11221
2017-01-24 22:12:20,599 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11224:Leader@356] - LEADING - LEADER ELECTION TOOK - 247
2017-01-24 22:12:20,600 [myid:0] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11221:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:58471
2017-01-24 22:12:20,600 [myid:0] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11221:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:58471
2017-01-24 22:12:20,601 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11221:Follower@64] - FOLLOWING - LEADER ELECTION TOOK - 247
2017-01-24 22:12:20,604 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11224:FileTxnSnapLog@281] - Snapshotting: 0x0 to /data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test4158500655544466289.junit.dir/data/version-2/snapshot.0
2017-01-24 22:12:20,608 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11221:SaslQuorumAuthLearner@79] - Skipping SASL authentication as quorum.auth.learnerRequireSasl=false
2017-01-24 22:12:20,609 [myid:0] - INFO [Thread-7:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:58471 (no session established for client)
2017-01-24 22:12:20,619 [myid:1] - INFO [LearnerHandler-/127.0.0.1:57759:LearnerHandler@287] - Follower sid: 0 : info : org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@2b6c7d59
2017-01-24 22:12:20,621 [myid:1] - INFO [LearnerHandler-/127.0.0.1:57759:LearnerHandler@342] - Synchronizing with Follower sid: 0 maxCommittedLog=0x0 minCommittedLog=0x0 peerLastZxid=0x0
2017-01-24 22:12:20,621 [myid:1] - INFO [LearnerHandler-/127.0.0.1:57759:LearnerHandler@441] - Sending snapshot last zxid of peer is 0x0 zxid of leader is 0x100000000sent zxid of db as 0x0
2017-01-24 22:12:20,622 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11221:Learner@329] - Getting a snapshot from leader
2017-01-24 22:12:20,626 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11221:FileTxnSnapLog@281] - Snapshotting: 0x0 to /data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test8717551953335097579.junit.dir/data/version-2/snapshot.0
2017-01-24 22:12:20,627 [myid:1] - INFO [LearnerHandler-/127.0.0.1:57759:LearnerHandler@477] - Received NEWLEADER-ACK message from 0
2017-01-24 22:12:20,627 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11224:Leader@934] - Have quorum of supporters, sids: [ 0,1 ]; starting up and setting last processed zxid: 0x100000000
2017-01-24 22:12:20,860 [myid:] - INFO [Thread-0:FourLetterWordMain@43] - connecting to 127.0.0.1 11221
2017-01-24 22:12:20,860 [myid:0] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11221:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:58473
2017-01-24 22:12:20,861 [myid:0] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11221:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:58473
2017-01-24 22:12:20,862 [myid:0] - INFO [Thread-10:NIOServerCnxn$StatCommand@655] - Stat command output
2017-01-24 22:12:20,863 [myid:0] - INFO [Thread-10:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:58473 (no session established for client)
2017-01-24 22:12:20,863 [myid:] - INFO [Thread-0:FourLetterWordMain@43] - connecting to 127.0.0.1 11224
2017-01-24 22:12:20,864 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11224:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:36288
2017-01-24 22:12:20,864 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11224:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:36288
2017-01-24 22:12:20,864 [myid:1] - INFO [Thread-11:NIOServerCnxn$StatCommand@655] - Stat command output
2017-01-24 22:12:20,865 [myid:1] - INFO [Thread-11:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:36288 (no session established for client)
2017-01-24 22:12:20,869 [myid:] - INFO [Thread-0:Environment@100] - Client environment:zookeeper.version=3.4.5--1, built on 01/25/2017 06:01 GMT
2017-01-24 22:12:20,869 [myid:] - INFO [Thread-0:Environment@100] - Client environment:host.name=ec2-unittest-ub12-m3-large-0e11.vpc.cloudera.com
2017-01-24 22:12:20,869 [myid:] - INFO [Thread-0:Environment@100] - Client environment:java.version=1.7.0_75
2017-01-24 22:12:20,870 [myid:] - INFO [Thread-0:Environment@100] - Client environment:java.vendor=Oracle Corporation
2017-01-24 22:12:20,870 [myid:] - INFO [Thread-0:Environment@100] - Client environment:java.home=/data/jenkins/tools/hudson.model.JDK/Java_7/jre
2017-01-24 22:12:20,870 [myid:] - INFO [Thread-0:Environment@100] - Client environment:java.class.path=/data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/classes:/data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/lib/antlr-2.7.6.jar:/data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/lib/checkstyle-5.0.jar:/data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/lib/commons-beanutils-core-1.7.0.jar:/data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/lib/commons-cli-1.0.jar:/data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/lib/commons-collections-3.2.2.jar:/data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/lib/commons-io-2.4.jar:/data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/lib/commons-lang-1.0.jar:/data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/lib/commons-logging-1.0.3.jar:/data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/lib/google-collections-0.9.jar:/data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/lib/junit-4.8.1.jar:/data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/lib/kerb-admin-1.0.0-RC2.jar:/data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/lib/kerb-client-1.0.0-RC2.jar:/data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/lib/kerb-common-1.0.0-RC2.jar:/data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/lib/kerb-core-1.0.0-RC2.jar:/data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/lib/kerb-crypto-1.0.0-RC2.jar:/data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/lib/kerb-identity-1.0.0-RC2.jar:/data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/lib/kerb-server-1.0.0-RC2.jar:/data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/lib/kerb-simplekdc-1.0.0-RC2.jar:/data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/lib/kerb-util-1.0.0-RC2.jar:/data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/lib/kerby-asn1-1.0.0-RC2.jar:/data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/lib/kerby-config-1.0.0-RC2.jar:/data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/lib/kerby-pkix-1.0.0-RC2.jar:/data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/lib/kerby-util-1.0.0-RC2.jar:/data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/lib/log4j-1.2.17.jar:/data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/lib/mockito-all-1.8.2.jar:/data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/lib/slf4j-api-1.7.14.jar:/data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/lib/slf4j-log4j12-1.7.14.jar:/data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/classes:/data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/src/java/lib/ivy-2.2.0.jar:/data/jenkins/tools/hudson.tasks.Ant_AntInstallation/Ant_1.8.2/apache-ant-1.9.6/lib/ant.jar:/data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/lib/jline-2.11.jar:/data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/lib/log4j-1.2.16.jar:/data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/lib/netty-3.10.5.Final.jar:/data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/lib/slf4j-api-1.7.5.jar:/data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/lib/slf4j-log4j12-1.7.5.jar:/mnt/toolchain/clover-ant-4.0.3/lib/clover.jar:/data/jenkins/tools/hudson.tasks.Ant_AntInstallation/Ant_1.8.2/apache-ant-1.9.6/lib/ant-launcher.jar:/data/jenkins/tools/hudson.tasks.Ant_AntInstallation/Ant_1.8.2/apache-ant-1.9.6/lib/ant-junit.jar:/data/jenkins/tools/hudson.tasks.Ant_AntInstallation/Ant_1.8.2/apache-ant-1.9.6/lib/ant-junit4.jar
2017-01-24 22:12:20,870 [myid:] - INFO [Thread-0:Environment@100] - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2017-01-24 22:12:20,870 [myid:] - INFO [Thread-0:Environment@100] - Client environment:java.io.tmpdir=/tmp
2017-01-24 22:12:20,870 [myid:] - INFO [Thread-0:Environment@100] - Client environment:java.compiler=<NA>
2017-01-24 22:12:20,870 [myid:] - INFO [Thread-0:Environment@100] - Client environment:os.name=Linux
2017-01-24 22:12:20,870 [myid:] - INFO [Thread-0:Environment@100] - Client environment:os.arch=amd64
2017-01-24 22:12:20,870 [myid:] - INFO [Thread-0:Environment@100] - Client environment:os.version=3.2.0-33-virtual
2017-01-24 22:12:20,870 [myid:] - INFO [Thread-0:Environment@100] - Client environment:user.name=jenkins
2017-01-24 22:12:20,870 [myid:] - INFO [Thread-0:Environment@100] - Client environment:user.home=/var/lib/jenkins
2017-01-24 22:12:20,871 [myid:] - INFO [Thread-0:Environment@100] - Client environment:user.dir=/data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7
2017-01-24 22:12:20,871 [myid:] - INFO [Thread-0:ZooKeeper@438] - Initiating client connection, connectString=127.0.0.1:11221,127.0.0.1:11224 sessionTimeout=30000 watcher=org.apache.zookeeper.test.ClientBase$CountdownWatcher@693a2c6a
2017-01-24 22:12:20,889 [myid:] - WARN [Thread-0-SendThread(localhost:11221):ClientCnxn$SendThread@957] - SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: '/data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test2999887027410032136.junit.dir/jaas.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.
2017-01-24 22:12:20,891 [myid:] - INFO [Thread-0-SendThread(localhost:11221):ClientCnxn$SendThread@975] - Opening socket connection to server localhost/127.0.0.1:11221
2017-01-24 22:12:20,891 [myid:0] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11221:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:58475
2017-01-24 22:12:20,891 [myid:] - INFO [Thread-0-SendThread(localhost:11221):ClientCnxn$SendThread@852] - Socket connection established, initiating session, client: /127.0.0.1:58475, server: localhost/127.0.0.1:11221
2017-01-24 22:12:20,893 [myid:0] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11221:ZooKeeperServer@839] - Client attempting to establish new session at /127.0.0.1:58475
2017-01-24 22:12:20,896 [myid:1] - INFO [SyncThread:1:FileTxnLog@199] - Creating new log file: log.100000001
2017-01-24 22:12:20,897 [myid:0] - WARN [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11221:Follower@119] - Got zxid 0x100000001 expected 0x1
2017-01-24 22:12:20,897 [myid:0] - INFO [SyncThread:0:FileTxnLog@199] - Creating new log file: log.100000001
2017-01-24 22:12:20,906 [myid:0] - INFO [CommitProcessor:0:ZooKeeperServer@595] - Established session 0x59d440e8120000 with negotiated timeout 30000 for client /127.0.0.1:58475
2017-01-24 22:12:20,906 [myid:] - INFO [Thread-0-SendThread(localhost:11221):ClientCnxn$SendThread@1235] - Session establishment complete on server localhost/127.0.0.1:11221, sessionid = 0x59d440e8120000, negotiated timeout = 30000
2017-01-24 22:12:20,922 [myid:1] - INFO [ProcessThread(sid:1 cport:-1)::PrepRequestProcessor@494] - Processed session termination for sessionid: 0x59d440e8120000
2017-01-24 22:12:20,925 [myid:0] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11221:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:58475 which had sessionid 0x59d440e8120000
2017-01-24 22:12:20,925 [myid:] - INFO [Thread-0:ZooKeeper@684] - Session: 0x59d440e8120000 closed
2017-01-24 22:12:20,926 [myid:] - INFO [Thread-0:JUnit4ZKTestRunner$LoggedInvokeMethod@55] - Memory used 14639
2017-01-24 22:12:20,926 [myid:] - INFO [Thread-0:JUnit4ZKTestRunner$LoggedInvokeMethod@60] - Number of threads 39
2017-01-24 22:12:20,926 [myid:] - INFO [Thread-0:JUnit4ZKTestRunner$LoggedInvokeMethod@65] - FINISHED TEST METHOD testAuthLearnerAgainstNoAuthRequiredServer
2017-01-24 22:12:20,926 [myid:] - INFO [Thread-0-EventThread:ClientCnxn$EventThread@512] - EventThread shut down
2017-01-24 22:12:20,927 [myid:] - INFO [main:QuorumBase@314] - Shutting down quorum peer QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11221
2017-01-24 22:12:20,927 [myid:] - INFO [main:Follower@167] - shutdown called
java.lang.Exception: shutdown Follower
at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:167)
at org.apache.zookeeper.server.quorum.QuorumPeer.shutdown(QuorumPeer.java:896)
at org.apache.zookeeper.test.QuorumBase.shutdown(QuorumBase.java:315)
at org.apache.zookeeper.server.quorum.QuorumPeerTestBase$TestQPMain.shutdown(QuorumPeerTestBase.java:59)
at org.apache.zookeeper.server.quorum.QuorumPeerTestBase$MainThread.shutdown(QuorumPeerTestBase.java:152)
at org.apache.zookeeper.server.quorum.auth.QuorumAuthTestBase.shutdown(QuorumAuthTestBase.java:138)
at org.apache.zookeeper.server.quorum.auth.QuorumAuthTestBase.shutdownAll(QuorumAuthTestBase.java:131)
at org.apache.zookeeper.server.quorum.auth.QuorumAuthUpgradeTest.tearDown(QuorumAuthUpgradeTest.java:68)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:37)
at org.junit.rules.TestWatchman$1.evaluate(TestWatchman.java:48)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:76)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
at org.junit.runners.ParentRunner.run(ParentRunner.java:236)
at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:39)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:535)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1182)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:1033)
2017-01-24 22:12:20,928 [myid:] - INFO [main:FollowerZooKeeperServer@139] - Shutting down
2017-01-24 22:12:20,928 [myid:] - INFO [main:ZooKeeperServer@419] - shutting down
2017-01-24 22:12:20,928 [myid:] - INFO [main:FollowerRequestProcessor@105] - Shutting down
2017-01-24 22:12:20,928 [myid:] - INFO [main:CommitProcessor@181] - Shutting down
2017-01-24 22:12:20,928 [myid:0] - INFO [FollowerRequestProcessor:0:FollowerRequestProcessor@95] - FollowerRequestProcessor exited loop!
2017-01-24 22:12:20,929 [myid:] - INFO [main:FinalRequestProcessor@415] - shutdown of request processor complete
2017-01-24 22:12:20,929 [myid:0] - INFO [CommitProcessor:0:CommitProcessor@150] - CommitProcessor exited loop!
2017-01-24 22:12:20,929 [myid:] - INFO [main:SyncRequestProcessor@175] - Shutting down
2017-01-24 22:12:20,929 [myid:0] - INFO [SyncThread:0:SyncRequestProcessor@155] - SyncRequestProcessor exited!
2017-01-24 22:12:20,930 [myid:0] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11221:NIOServerCnxnFactory@224] - NIOServerCnxn factory exited run method
2017-01-24 22:12:20,930 [myid:0] - ERROR [localhost/127.0.0.1:11223:QuorumCnxManager$Listener@715] - Exception while listening
java.net.SocketException: Socket closed
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
at java.net.ServerSocket.implAccept(ServerSocket.java:530)
at java.net.ServerSocket.accept(ServerSocket.java:498)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$Listener.run(QuorumCnxManager.java:696)
2017-01-24 22:12:20,930 [myid:1] - WARN [RecvWorker:0:QuorumCnxManager$RecvWorker@980] - Connection broken for id 0, my id = 1, error =
java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:392)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:965)
2017-01-24 22:12:20,930 [myid:1] - WARN [RecvWorker:0:QuorumCnxManager$RecvWorker@983] - Interrupting SendWorker
2017-01-24 22:12:20,931 [myid:1] - WARN [SendWorker:0:QuorumCnxManager$SendWorker@896] - Interrupted while waiting for message on queue
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1049)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:73)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:884)
2017-01-24 22:12:20,931 [myid:1] - WARN [SendWorker:0:QuorumCnxManager$SendWorker@906] - Send worker leaving thread
2017-01-24 22:12:20,931 [myid:0] - WARN [SendWorker:1:QuorumCnxManager$SendWorker@896] - Interrupted while waiting for message on queue
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1049)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:73)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:884)
2017-01-24 22:12:20,932 [myid:0] - WARN [SendWorker:1:QuorumCnxManager$SendWorker@906] - Send worker leaving thread
2017-01-24 22:12:20,931 [myid:0] - WARN [RecvWorker:1:QuorumCnxManager$RecvWorker@980] - Connection broken for id 1, my id = 0, error =
java.net.SocketException: Socket closed
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at java.io.BufferedInputStream.read(BufferedInputStream.java:254)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:965)
2017-01-24 22:12:20,932 [myid:0] - WARN [RecvWorker:1:QuorumCnxManager$RecvWorker@983] - Interrupting SendWorker
2017-01-24 22:12:20,933 [myid:] - INFO [main:QuorumBase@318] - Shutting down leader election QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11221
2017-01-24 22:12:20,933 [myid:] - INFO [main:QuorumBase@323] - Waiting for QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11221 to exit thread
2017-01-24 22:12:21,930 [myid:0] - INFO [localhost/127.0.0.1:11223:QuorumCnxManager$Listener@728] - Leaving listener
2017-01-24 22:12:22,648 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11221:Follower@167] - shutdown called
java.lang.Exception: shutdown Follower
at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:167)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:855)
2017-01-24 22:12:22,648 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11221:FollowerZooKeeperServer@139] - Shutting down
2017-01-24 22:12:22,648 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11221:ZooKeeperServer@419] - shutting down
2017-01-24 22:12:22,648 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11221:FollowerRequestProcessor@105] - Shutting down
2017-01-24 22:12:22,648 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11221:CommitProcessor@181] - Shutting down
2017-01-24 22:12:22,648 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11221:FinalRequestProcessor@415] - shutdown of request processor complete
2017-01-24 22:12:22,648 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11221:SyncRequestProcessor@175] - Shutting down
2017-01-24 22:12:22,648 [myid:0] - WARN [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11221:QuorumPeer@879] - QuorumPeer main thread exited
2017-01-24 22:12:22,651 [myid:] - INFO [main:ZKTestCase$1@60] - SUCCEEDED testAuthLearnerAgainstNoAuthRequiredServer
2017-01-24 22:12:22,651 [myid:] - INFO [main:ZKTestCase$1@55] - FINISHED testAuthLearnerAgainstNoAuthRequiredServer
2017-01-24 22:12:22,652 [myid:] - INFO [main:ZKTestCase$1@50] - STARTING testRollingUpgrade
2017-01-24 22:12:22,653 [myid:] - INFO [Thread-12:JUnit4ZKTestRunner$LoggedInvokeMethod@50] - RUNNING TEST METHOD testRollingUpgrade
2017-01-24 22:12:22,653 [myid:] - INFO [Thread-12:PortAssignment@32] - assigning port 11227
2017-01-24 22:12:22,653 [myid:] - INFO [Thread-12:PortAssignment@32] - assigning port 11228
2017-01-24 22:12:22,653 [myid:] - INFO [Thread-12:PortAssignment@32] - assigning port 11229
2017-01-24 22:12:22,653 [myid:] - INFO [Thread-12:PortAssignment@32] - assigning port 11230
2017-01-24 22:12:22,653 [myid:] - INFO [Thread-12:PortAssignment@32] - assigning port 11231
2017-01-24 22:12:22,653 [myid:] - INFO [Thread-12:PortAssignment@32] - assigning port 11232
2017-01-24 22:12:22,653 [myid:] - INFO [Thread-12:PortAssignment@32] - assigning port 11233
2017-01-24 22:12:22,654 [myid:] - INFO [Thread-12:PortAssignment@32] - assigning port 11234
2017-01-24 22:12:22,654 [myid:] - INFO [Thread-12:PortAssignment@32] - assigning port 11235
2017-01-24 22:12:22,654 [myid:] - INFO [Thread-12:QuorumPeerTestBase$MainThread@81] - id = 0 tmpDir = /data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test1594648493062198412.junit.dir clientPort = 11227
2017-01-24 22:12:22,655 [myid:] - INFO [Thread-13:QuorumPeerConfig@111] - Reading configuration from: /data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test1594648493062198412.junit.dir/zoo.cfg
2017-01-24 22:12:22,655 [myid:] - INFO [Thread-12:QuorumPeerTestBase$MainThread@81] - id = 1 tmpDir = /data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test8525006967275020093.junit.dir clientPort = 11230
2017-01-24 22:12:22,655 [myid:] - INFO [Thread-13:QuorumPeerConfig@374] - Defaulting to majority quorums
2017-01-24 22:12:22,656 [myid:0] - INFO [Thread-13:DatadirCleanupManager@78] - autopurge.snapRetainCount set to 3
2017-01-24 22:12:22,656 [myid:0] - INFO [Thread-13:DatadirCleanupManager@79] - autopurge.purgeInterval set to 0
2017-01-24 22:12:22,656 [myid:0] - INFO [Thread-13:DatadirCleanupManager@101] - Purge task is not scheduled.
2017-01-24 22:12:22,656 [myid:] - INFO [Thread-12:QuorumPeerTestBase$MainThread@81] - id = 2 tmpDir = /data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test2781541844609161589.junit.dir clientPort = 11233
2017-01-24 22:12:22,656 [myid:0] - WARN [Thread-13:QuorumPeerMain@129] - Unable to register log4j JMX control
javax.management.InstanceAlreadyExistsException: log4j:hiearchy=default
at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:437)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1898)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:966)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324)
at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
at org.apache.zookeeper.jmx.ManagedUtil.registerLog4jMBeans(ManagedUtil.java:53)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:127)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:116)
at org.apache.zookeeper.server.quorum.QuorumPeerTestBase$MainThread.run(QuorumPeerTestBase.java:140)
at java.lang.Thread.run(Thread.java:745)
2017-01-24 22:12:22,657 [myid:0] - INFO [Thread-13:QuorumPeerMain@132] - Starting quorum peer
2017-01-24 22:12:22,657 [myid:] - INFO [Thread-15:QuorumPeerConfig@111] - Reading configuration from: /data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test2781541844609161589.junit.dir/zoo.cfg
2017-01-24 22:12:22,657 [myid:0] - INFO [Thread-13:NIOServerCnxnFactory@94] - binding to port 0.0.0.0/0.0.0.0:11227
2017-01-24 22:12:22,657 [myid:] - INFO [Thread-15:QuorumPeerConfig@374] - Defaulting to majority quorums
2017-01-24 22:12:22,658 [myid:0] - INFO [Thread-13:QuorumPeer@1048] - minSessionTimeout set to -1
2017-01-24 22:12:22,658 [myid:0] - INFO [Thread-13:QuorumPeer@1059] - maxSessionTimeout set to -1
2017-01-24 22:12:22,656 [myid:] - INFO [Thread-14:QuorumPeerConfig@111] - Reading configuration from: /data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test8525006967275020093.junit.dir/zoo.cfg
2017-01-24 22:12:22,658 [myid:0] - INFO [Thread-13:QuorumPeer@1277] - QuorumPeer communication is not secured!
2017-01-24 22:12:22,658 [myid:0] - INFO [Thread-13:QuorumPeer@1306] - quorum.cnxn.threads.size set to 20
2017-01-24 22:12:22,658 [myid:] - INFO [Thread-14:QuorumPeerConfig@374] - Defaulting to majority quorums
2017-01-24 22:12:22,659 [myid:0] - INFO [Thread-13:QuorumPeer@540] - currentEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
2017-01-24 22:12:22,659 [myid:1] - INFO [Thread-14:DatadirCleanupManager@78] - autopurge.snapRetainCount set to 3
2017-01-24 22:12:22,659 [myid:1] - INFO [Thread-14:DatadirCleanupManager@79] - autopurge.purgeInterval set to 0
2017-01-24 22:12:22,659 [myid:1] - INFO [Thread-14:DatadirCleanupManager@101] - Purge task is not scheduled.
2017-01-24 22:12:22,659 [myid:1] - WARN [Thread-14:QuorumPeerMain@129] - Unable to register log4j JMX control
javax.management.InstanceAlreadyExistsException: log4j:hiearchy=default
at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:437)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1898)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:966)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324)
at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
at org.apache.zookeeper.jmx.ManagedUtil.registerLog4jMBeans(ManagedUtil.java:53)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:127)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:116)
at org.apache.zookeeper.server.quorum.QuorumPeerTestBase$MainThread.run(QuorumPeerTestBase.java:140)
at java.lang.Thread.run(Thread.java:745)
2017-01-24 22:12:22,660 [myid:1] - INFO [Thread-14:QuorumPeerMain@132] - Starting quorum peer
2017-01-24 22:12:22,660 [myid:1] - INFO [Thread-14:NIOServerCnxnFactory@94] - binding to port 0.0.0.0/0.0.0.0:11230
2017-01-24 22:12:22,660 [myid:1] - INFO [Thread-14:QuorumPeer@1048] - minSessionTimeout set to -1
2017-01-24 22:12:22,660 [myid:1] - INFO [Thread-14:QuorumPeer@1059] - maxSessionTimeout set to -1
2017-01-24 22:12:22,660 [myid:1] - INFO [Thread-14:QuorumPeer@1277] - QuorumPeer communication is not secured!
2017-01-24 22:12:22,661 [myid:1] - INFO [Thread-14:QuorumPeer@1306] - quorum.cnxn.threads.size set to 20
2017-01-24 22:12:22,661 [myid:1] - INFO [Thread-14:QuorumPeer@540] - currentEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
2017-01-24 22:12:22,658 [myid:2] - INFO [Thread-15:DatadirCleanupManager@78] - autopurge.snapRetainCount set to 3
2017-01-24 22:12:22,661 [myid:2] - INFO [Thread-15:DatadirCleanupManager@79] - autopurge.purgeInterval set to 0
2017-01-24 22:12:22,661 [myid:2] - INFO [Thread-15:DatadirCleanupManager@101] - Purge task is not scheduled.
2017-01-24 22:12:22,657 [myid:] - INFO [Thread-12:FourLetterWordMain@43] - connecting to 127.0.0.1 11227
2017-01-24 22:12:22,662 [myid:2] - WARN [Thread-15:QuorumPeerMain@129] - Unable to register log4j JMX control
javax.management.InstanceAlreadyExistsException: log4j:hiearchy=default
at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:437)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1898)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:966)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324)
at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
at org.apache.zookeeper.jmx.ManagedUtil.registerLog4jMBeans(ManagedUtil.java:53)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:127)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:116)
at org.apache.zookeeper.server.quorum.QuorumPeerTestBase$MainThread.run(QuorumPeerTestBase.java:140)
at java.lang.Thread.run(Thread.java:745)
2017-01-24 22:12:22,662 [myid:2] - INFO [Thread-15:QuorumPeerMain@132] - Starting quorum peer
2017-01-24 22:12:22,660 [myid:0] - INFO [Thread-13:QuorumPeer@555] - acceptedEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
2017-01-24 22:12:22,663 [myid:2] - INFO [Thread-15:NIOServerCnxnFactory@94] - binding to port 0.0.0.0/0.0.0.0:11233
2017-01-24 22:12:22,663 [myid:2] - INFO [Thread-15:QuorumPeer@1048] - minSessionTimeout set to -1
2017-01-24 22:12:22,663 [myid:2] - INFO [Thread-15:QuorumPeer@1059] - maxSessionTimeout set to -1
2017-01-24 22:12:22,663 [myid:2] - INFO [Thread-15:QuorumPeer@1277] - QuorumPeer communication is not secured!
2017-01-24 22:12:22,663 [myid:2] - INFO [Thread-15:QuorumPeer@1306] - quorum.cnxn.threads.size set to 20
2017-01-24 22:12:22,664 [myid:0] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11227:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:42807
2017-01-24 22:12:22,664 [myid:2] - INFO [Thread-15:QuorumPeer@540] - currentEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
2017-01-24 22:12:22,664 [myid:0] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11227:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:42807
2017-01-24 22:12:22,665 [myid:0] - INFO [Thread-16:QuorumCnxManager$Listener@691] - My election bind port: 0.0.0.0/0.0.0.0:11229
2017-01-24 22:12:22,665 [myid:1] - INFO [Thread-14:QuorumPeer@555] - acceptedEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
2017-01-24 22:12:22,666 [myid:0] - INFO [Thread-17:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:42807 (no session established for client)
2017-01-24 22:12:22,666 [myid:2] - INFO [Thread-15:QuorumPeer@555] - acceptedEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
2017-01-24 22:12:22,667 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11227:QuorumPeer@781] - LOOKING
2017-01-24 22:12:22,667 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11227:FastLeaderElection@744] - New election. My id = 0, proposed zxid=0x0
2017-01-24 22:12:22,668 [myid:0] - INFO [WorkerReceiver[myid=0]:FastLeaderElection@542] - Notification: 0 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 0 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)
2017-01-24 22:12:22,668 [myid:0] - WARN [WorkerSender[myid=0]:QuorumCnxManager@559] - Cannot open channel to 1 at election address localhost/127.0.0.1:11232
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:538)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:514)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:393)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:365)
at java.lang.Thread.run(Thread.java:745)
2017-01-24 22:12:22,669 [myid:0] - WARN [WorkerSender[myid=0]:QuorumCnxManager@559] - Cannot open channel to 2 at election address localhost/127.0.0.1:11235
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:538)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:514)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:393)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:365)
at java.lang.Thread.run(Thread.java:745)
2017-01-24 22:12:22,670 [myid:1] - INFO [Thread-18:QuorumCnxManager$Listener@691] - My election bind port: 0.0.0.0/0.0.0.0:11232
2017-01-24 22:12:22,670 [myid:2] - INFO [Thread-19:QuorumCnxManager$Listener@691] - My election bind port: 0.0.0.0/0.0.0.0:11235
2017-01-24 22:12:22,672 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11230:QuorumPeer@781] - LOOKING
2017-01-24 22:12:22,672 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11233:QuorumPeer@781] - LOOKING
2017-01-24 22:12:22,672 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11230:FastLeaderElection@744] - New election. My id = 1, proposed zxid=0x0
2017-01-24 22:12:22,673 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11233:FastLeaderElection@744] - New election. My id = 2, proposed zxid=0x0
2017-01-24 22:12:22,674 [myid:0] - INFO [localhost/127.0.0.1:11229:QuorumCnxManager$Listener@698] - Received connection request /127.0.0.1:51824
2017-01-24 22:12:22,680 [myid:1] - INFO [localhost/127.0.0.1:11232:QuorumCnxManager$Listener@698] - Received connection request /127.0.0.1:45173
2017-01-24 22:12:22,681 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@542] - Notification: 1 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)
2017-01-24 22:12:22,682 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@542] - Notification: 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)
2017-01-24 22:12:22,683 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@542] - Notification: 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)
2017-01-24 22:12:22,683 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@542] - Notification: 0 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 0 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)
2017-01-24 22:12:22,683 [myid:0] - INFO [localhost/127.0.0.1:11229:QuorumCnxManager$Listener@698] - Received connection request /127.0.0.1:51825
2017-01-24 22:12:22,683 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@542] - Notification: 1 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)
2017-01-24 22:12:22,684 [myid:0] - INFO [WorkerReceiver[myid=0]:FastLeaderElection@542] - Notification: 1 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)
2017-01-24 22:12:22,684 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@542] - Notification: 0 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 0 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)
2017-01-24 22:12:22,685 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@542] - Notification: 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)
2017-01-24 22:12:22,685 [myid:0] - INFO [WorkerReceiver[myid=0]:FastLeaderElection@542] - Notification: 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)
2017-01-24 22:12:22,685 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@542] - Notification: 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)
2017-01-24 22:12:22,686 [myid:0] - INFO [WorkerReceiver[myid=0]:FastLeaderElection@542] - Notification: 1 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 0 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)
2017-01-24 22:12:22,686 [myid:0] - INFO [WorkerReceiver[myid=0]:FastLeaderElection@542] - Notification: 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 0 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)
2017-01-24 22:12:22,686 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@542] - Notification: 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 0 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)
2017-01-24 22:12:22,686 [myid:0] - INFO [WorkerReceiver[myid=0]:FastLeaderElection@542] - Notification: 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)
2017-01-24 22:12:22,687 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@542] - Notification: 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 0 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)
2017-01-24 22:12:22,887 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11233:QuorumPeer@861] - LEADING
2017-01-24 22:12:22,887 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11227:QuorumPeer@849] - FOLLOWING
2017-01-24 22:12:22,887 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11227:ZooKeeperServer@162] - Created server with tickTime 4000 minSessionTimeout 8000 maxSessionTimeout 80000 datadir /data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test1594648493062198412.junit.dir/data/version-2 snapdir /data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test1594648493062198412.junit.dir/data/version-2
2017-01-24 22:12:22,887 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11233:ZooKeeperServer@162] - Created server with tickTime 4000 minSessionTimeout 8000 maxSessionTimeout 80000 datadir /data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test2781541844609161589.junit.dir/data/version-2 snapdir /data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test2781541844609161589.junit.dir/data/version-2
2017-01-24 22:12:22,887 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11227:Follower@64] - FOLLOWING - LEADER ELECTION TOOK - 220
2017-01-24 22:12:22,887 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11233:Leader@356] - LEADING - LEADER ELECTION TOOK - 214
2017-01-24 22:12:22,888 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11233:FileTxnSnapLog@281] - Snapshotting: 0x0 to /data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test2781541844609161589.junit.dir/data/version-2/snapshot.0
2017-01-24 22:12:22,887 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11230:QuorumPeer@849] - FOLLOWING
2017-01-24 22:12:22,889 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11230:ZooKeeperServer@162] - Created server with tickTime 4000 minSessionTimeout 8000 maxSessionTimeout 80000 datadir /data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test8525006967275020093.junit.dir/data/version-2 snapdir /data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test8525006967275020093.junit.dir/data/version-2
2017-01-24 22:12:22,889 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11230:Follower@64] - FOLLOWING - LEADER ELECTION TOOK - 217
2017-01-24 22:12:22,890 [myid:2] - INFO [LearnerHandler-/127.0.0.1:42553:LearnerHandler@287] - Follower sid: 0 : info : org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@559c4c8e
2017-01-24 22:12:22,890 [myid:2] - INFO [LearnerHandler-/127.0.0.1:42554:LearnerHandler@287] - Follower sid: 1 : info : org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@4f1e5ed9
2017-01-24 22:12:22,892 [myid:2] - INFO [LearnerHandler-/127.0.0.1:42553:LearnerHandler@342] - Synchronizing with Follower sid: 0 maxCommittedLog=0x0 minCommittedLog=0x0 peerLastZxid=0x0
2017-01-24 22:12:22,892 [myid:2] - INFO [LearnerHandler-/127.0.0.1:42554:LearnerHandler@342] - Synchronizing with Follower sid: 1 maxCommittedLog=0x0 minCommittedLog=0x0 peerLastZxid=0x0
2017-01-24 22:12:22,892 [myid:2] - INFO [LearnerHandler-/127.0.0.1:42553:LearnerHandler@441] - Sending snapshot last zxid of peer is 0x0 zxid of leader is 0x100000000sent zxid of db as 0x0
2017-01-24 22:12:22,892 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11227:Learner@329] - Getting a snapshot from leader
2017-01-24 22:12:22,892 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11230:Learner@329] - Getting a snapshot from leader
2017-01-24 22:12:22,893 [myid:2] - INFO [LearnerHandler-/127.0.0.1:42554:LearnerHandler@441] - Sending snapshot last zxid of peer is 0x0 zxid of leader is 0x100000000sent zxid of db as 0x0
2017-01-24 22:12:22,894 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11227:FileTxnSnapLog@281] - Snapshotting: 0x0 to /data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test1594648493062198412.junit.dir/data/version-2/snapshot.0
2017-01-24 22:12:22,894 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11230:FileTxnSnapLog@281] - Snapshotting: 0x0 to /data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test8525006967275020093.junit.dir/data/version-2/snapshot.0
2017-01-24 22:12:22,895 [myid:2] - INFO [LearnerHandler-/127.0.0.1:42553:LearnerHandler@477] - Received NEWLEADER-ACK message from 0
2017-01-24 22:12:22,895 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11233:Leader@934] - Have quorum of supporters, sids: [ 0,2 ]; starting up and setting last processed zxid: 0x100000000
2017-01-24 22:12:22,895 [myid:2] - INFO [LearnerHandler-/127.0.0.1:42554:LearnerHandler@477] - Received NEWLEADER-ACK message from 1
2017-01-24 22:12:22,917 [myid:] - INFO [Thread-12:FourLetterWordMain@43] - connecting to 127.0.0.1 11227
2017-01-24 22:12:22,918 [myid:0] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11227:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:42815
2017-01-24 22:12:22,918 [myid:0] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11227:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:42815
2017-01-24 22:12:22,918 [myid:0] - INFO [Thread-23:NIOServerCnxn$StatCommand@655] - Stat command output
2017-01-24 22:12:22,919 [myid:0] - INFO [Thread-23:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:42815 (no session established for client)
2017-01-24 22:12:22,919 [myid:] - INFO [Thread-12:FourLetterWordMain@43] - connecting to 127.0.0.1 11230
2017-01-24 22:12:22,920 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11230:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:52317
2017-01-24 22:12:22,920 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11230:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:52317
2017-01-24 22:12:22,921 [myid:1] - INFO [Thread-24:NIOServerCnxn$StatCommand@655] - Stat command output
2017-01-24 22:12:22,921 [myid:1] - INFO [Thread-24:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:52317 (no session established for client)
2017-01-24 22:12:22,921 [myid:] - INFO [Thread-12:FourLetterWordMain@43] - connecting to 127.0.0.1 11233
2017-01-24 22:12:22,922 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45787
2017-01-24 22:12:22,923 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:45787
2017-01-24 22:12:22,923 [myid:2] - INFO [Thread-25:NIOServerCnxn$StatCommand@655] - Stat command output
2017-01-24 22:12:22,924 [myid:2] - INFO [Thread-25:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45787 (no session established for client)
2017-01-24 22:12:22,924 [myid:] - INFO [Thread-12:ZooKeeper@438] - Initiating client connection, connectString=127.0.0.1:11227,127.0.0.1:11230,127.0.0.1:11233 sessionTimeout=30000 watcher=org.apache.zookeeper.test.ClientBase$CountdownWatcher@36c0ae96
2017-01-24 22:12:22,925 [myid:] - WARN [Thread-12-SendThread(localhost:11230):ClientCnxn$SendThread@957] - SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: '/data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test2999887027410032136.junit.dir/jaas.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.
2017-01-24 22:12:22,925 [myid:] - INFO [Thread-12-SendThread(localhost:11230):ClientCnxn$SendThread@975] - Opening socket connection to server localhost/127.0.0.1:11230
2017-01-24 22:12:22,926 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11230:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:52319
2017-01-24 22:12:22,926 [myid:] - INFO [Thread-12-SendThread(localhost:11230):ClientCnxn$SendThread@852] - Socket connection established, initiating session, client: /127.0.0.1:52319, server: localhost/127.0.0.1:11230
2017-01-24 22:12:22,927 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11230:ZooKeeperServer@839] - Client attempting to establish new session at /127.0.0.1:52319
2017-01-24 22:12:22,930 [myid:1] - WARN [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11230:Follower@119] - Got zxid 0x100000001 expected 0x1
2017-01-24 22:12:22,930 [myid:0] - WARN [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11227:Follower@119] - Got zxid 0x100000001 expected 0x1
2017-01-24 22:12:22,930 [myid:2] - INFO [SyncThread:2:FileTxnLog@199] - Creating new log file: log.100000001
2017-01-24 22:12:22,931 [myid:1] - INFO [SyncThread:1:FileTxnLog@199] - Creating new log file: log.100000001
2017-01-24 22:12:22,931 [myid:0] - INFO [SyncThread:0:FileTxnLog@199] - Creating new log file: log.100000001
2017-01-24 22:12:22,933 [myid:1] - INFO [CommitProcessor:1:ZooKeeperServer@595] - Established session 0x159d440f0ed0000 with negotiated timeout 30000 for client /127.0.0.1:52319
2017-01-24 22:12:22,933 [myid:] - INFO [Thread-12-SendThread(localhost:11230):ClientCnxn$SendThread@1235] - Session establishment complete on server localhost/127.0.0.1:11230, sessionid = 0x159d440f0ed0000, negotiated timeout = 30000
2017-01-24 22:12:22,937 [myid:] - INFO [Thread-12:QuorumAuthUpgradeTest@229] - Restarting server myid=0
2017-01-24 22:12:22,937 [myid:] - INFO [Thread-12:QuorumBase@314] - Shutting down quorum peer QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11227
2017-01-24 22:12:22,937 [myid:] - INFO [Thread-12:Follower@167] - shutdown called
java.lang.Exception: shutdown Follower
at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:167)
at org.apache.zookeeper.server.quorum.QuorumPeer.shutdown(QuorumPeer.java:896)
at org.apache.zookeeper.test.QuorumBase.shutdown(QuorumBase.java:315)
at org.apache.zookeeper.server.quorum.QuorumPeerTestBase$TestQPMain.shutdown(QuorumPeerTestBase.java:59)
at org.apache.zookeeper.server.quorum.QuorumPeerTestBase$MainThread.shutdown(QuorumPeerTestBase.java:152)
at org.apache.zookeeper.server.quorum.auth.QuorumAuthTestBase.shutdown(QuorumAuthTestBase.java:138)
at org.apache.zookeeper.server.quorum.auth.QuorumAuthUpgradeTest.restartServer(QuorumAuthUpgradeTest.java:230)
at org.apache.zookeeper.server.quorum.auth.QuorumAuthUpgradeTest.testRollingUpgrade(QuorumAuthUpgradeTest.java:194)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:52)
at org.junit.internal.runners.statements.FailOnTimeout$1.run(FailOnTimeout.java:28)
2017-01-24 22:12:22,938 [myid:] - INFO [Thread-12:FollowerZooKeeperServer@139] - Shutting down
2017-01-24 22:12:22,938 [myid:] - INFO [Thread-12:ZooKeeperServer@419] - shutting down
2017-01-24 22:12:22,938 [myid:] - INFO [Thread-12:FollowerRequestProcessor@105] - Shutting down
2017-01-24 22:12:22,938 [myid:] - INFO [Thread-12:CommitProcessor@181] - Shutting down
2017-01-24 22:12:22,938 [myid:] - INFO [Thread-12:FinalRequestProcessor@415] - shutdown of request processor complete
2017-01-24 22:12:22,938 [myid:0] - INFO [FollowerRequestProcessor:0:FollowerRequestProcessor@95] - FollowerRequestProcessor exited loop!
2017-01-24 22:12:22,938 [myid:0] - INFO [CommitProcessor:0:CommitProcessor@150] - CommitProcessor exited loop!
2017-01-24 22:12:22,939 [myid:] - INFO [Thread-12:SyncRequestProcessor@175] - Shutting down
2017-01-24 22:12:22,939 [myid:0] - INFO [SyncThread:0:SyncRequestProcessor@155] - SyncRequestProcessor exited!
2017-01-24 22:12:22,939 [myid:0] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11227:NIOServerCnxnFactory@224] - NIOServerCnxn factory exited run method
2017-01-24 22:12:22,939 [myid:0] - ERROR [localhost/127.0.0.1:11229:QuorumCnxManager$Listener@715] - Exception while listening
java.net.SocketException: Socket closed
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
at java.net.ServerSocket.implAccept(ServerSocket.java:530)
at java.net.ServerSocket.accept(ServerSocket.java:498)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$Listener.run(QuorumCnxManager.java:696)
2017-01-24 22:12:22,939 [myid:1] - WARN [RecvWorker:0:QuorumCnxManager$RecvWorker@980] - Connection broken for id 0, my id = 1, error =
java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:392)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:965)
2017-01-24 22:12:22,940 [myid:1] - WARN [RecvWorker:0:QuorumCnxManager$RecvWorker@983] - Interrupting SendWorker
2017-01-24 22:12:22,940 [myid:2] - WARN [RecvWorker:0:QuorumCnxManager$RecvWorker@980] - Connection broken for id 0, my id = 2, error =
java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:392)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:965)
2017-01-24 22:12:22,940 [myid:0] - WARN [RecvWorker:1:QuorumCnxManager$RecvWorker@980] - Connection broken for id 1, my id = 0, error =
java.net.SocketException: Socket closed
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at java.io.BufferedInputStream.read(BufferedInputStream.java:254)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:965)
2017-01-24 22:12:22,941 [myid:0] - WARN [RecvWorker:1:QuorumCnxManager$RecvWorker@983] - Interrupting SendWorker
2017-01-24 22:12:22,940 [myid:0] - WARN [SendWorker:2:QuorumCnxManager$SendWorker@896] - Interrupted while waiting for message on queue
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1049)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:73)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:884)
2017-01-24 22:12:22,941 [myid:0] - WARN [SendWorker:2:QuorumCnxManager$SendWorker@906] - Send worker leaving thread
2017-01-24 22:12:22,940 [myid:0] - WARN [RecvWorker:2:QuorumCnxManager$RecvWorker@980] - Connection broken for id 2, my id = 0, error =
java.net.SocketException: Socket closed
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at java.io.BufferedInputStream.read(BufferedInputStream.java:254)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:965)
2017-01-24 22:12:22,941 [myid:0] - WARN [RecvWorker:2:QuorumCnxManager$RecvWorker@983] - Interrupting SendWorker
2017-01-24 22:12:22,941 [myid:] - INFO [Thread-12:QuorumBase@318] - Shutting down leader election QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11227
2017-01-24 22:12:22,941 [myid:0] - WARN [SendWorker:1:QuorumCnxManager$SendWorker@896] - Interrupted while waiting for message on queue
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1049)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:73)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:884)
2017-01-24 22:12:22,942 [myid:0] - WARN [SendWorker:1:QuorumCnxManager$SendWorker@906] - Send worker leaving thread
2017-01-24 22:12:22,940 [myid:1] - WARN [SendWorker:0:QuorumCnxManager$SendWorker@896] - Interrupted while waiting for message on queue
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1049)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:73)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:884)
2017-01-24 22:12:22,942 [myid:1] - WARN [SendWorker:0:QuorumCnxManager$SendWorker@906] - Send worker leaving thread
2017-01-24 22:12:22,940 [myid:2] - WARN [RecvWorker:0:QuorumCnxManager$RecvWorker@983] - Interrupting SendWorker
2017-01-24 22:12:22,942 [myid:] - INFO [Thread-12:QuorumBase@323] - Waiting for QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11227 to exit thread
2017-01-24 22:12:22,943 [myid:2] - WARN [SendWorker:0:QuorumCnxManager$SendWorker@896] - Interrupted while waiting for message on queue
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1049)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:73)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:884)
2017-01-24 22:12:22,943 [myid:2] - WARN [SendWorker:0:QuorumCnxManager$SendWorker@906] - Send worker leaving thread
2017-01-24 22:12:23,380 [myid:0] - INFO [WorkerSender[myid=0]:FastLeaderElection$Messenger$WorkerSender@370] - WorkerSender is down
2017-01-24 22:12:23,380 [myid:0] - INFO [WorkerReceiver[myid=0]:FastLeaderElection$Messenger$WorkerReceiver@340] - WorkerReceiver is down
2017-01-24 22:12:23,940 [myid:0] - INFO [localhost/127.0.0.1:11229:QuorumCnxManager$Listener@728] - Leaving listener
2017-01-24 22:12:24,897 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11227:Follower@167] - shutdown called
java.lang.Exception: shutdown Follower
at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:167)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:855)
2017-01-24 22:12:24,897 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11227:FollowerZooKeeperServer@139] - Shutting down
2017-01-24 22:12:24,897 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11227:ZooKeeperServer@419] - shutting down
2017-01-24 22:12:24,898 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11227:FollowerRequestProcessor@105] - Shutting down
2017-01-24 22:12:24,898 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11227:CommitProcessor@181] - Shutting down
2017-01-24 22:12:24,898 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11227:FinalRequestProcessor@415] - shutdown of request processor complete
2017-01-24 22:12:24,898 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11227:SyncRequestProcessor@175] - Shutting down
2017-01-24 22:12:24,898 [myid:0] - WARN [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11227:QuorumPeer@879] - QuorumPeer main thread exited
2017-01-24 22:12:24,901 [myid:] - INFO [Thread-12:QuorumPeerTestBase$MainThread@81] - id = 0 tmpDir = /data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test6261823493417515862.junit.dir clientPort = 11227
2017-01-24 22:12:24,901 [myid:] - INFO [Thread-12:FourLetterWordMain@43] - connecting to 127.0.0.1 11227
2017-01-24 22:12:24,901 [myid:] - INFO [Thread-26:QuorumPeerConfig@111] - Reading configuration from: /data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test6261823493417515862.junit.dir/zoo.cfg
2017-01-24 22:12:24,902 [myid:] - INFO [Thread-12:ClientBase@246] - server 127.0.0.1:11227 not up java.net.ConnectException: Connection refused
2017-01-24 22:12:24,902 [myid:] - INFO [Thread-26:QuorumPeerConfig@374] - Defaulting to majority quorums
2017-01-24 22:12:24,902 [myid:0] - INFO [Thread-26:DatadirCleanupManager@78] - autopurge.snapRetainCount set to 3
2017-01-24 22:12:24,902 [myid:0] - INFO [Thread-26:DatadirCleanupManager@79] - autopurge.purgeInterval set to 0
2017-01-24 22:12:24,902 [myid:0] - INFO [Thread-26:DatadirCleanupManager@101] - Purge task is not scheduled.
2017-01-24 22:12:24,903 [myid:0] - WARN [Thread-26:QuorumPeerMain@129] - Unable to register log4j JMX control
javax.management.InstanceAlreadyExistsException: log4j:hiearchy=default
at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:437)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1898)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:966)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324)
at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
at org.apache.zookeeper.jmx.ManagedUtil.registerLog4jMBeans(ManagedUtil.java:53)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:127)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:116)
at org.apache.zookeeper.server.quorum.QuorumPeerTestBase$MainThread.run(QuorumPeerTestBase.java:140)
at java.lang.Thread.run(Thread.java:745)
2017-01-24 22:12:24,903 [myid:0] - INFO [Thread-26:QuorumPeerMain@132] - Starting quorum peer
2017-01-24 22:12:24,903 [myid:0] - INFO [Thread-26:NIOServerCnxnFactory@94] - binding to port 0.0.0.0/0.0.0.0:11227
2017-01-24 22:12:24,904 [myid:0] - INFO [Thread-26:QuorumPeer@1048] - minSessionTimeout set to -1
2017-01-24 22:12:24,904 [myid:0] - INFO [Thread-26:QuorumPeer@1059] - maxSessionTimeout set to -1
2017-01-24 22:12:24,904 [myid:0] - INFO [Thread-26:QuorumPeer@1279] - quorum.auth.enableSasl set to true
2017-01-24 22:12:24,904 [myid:0] - INFO [Thread-26:QuorumPeer@1264] - quorum.auth.serverRequireSasl set to false
2017-01-24 22:12:24,904 [myid:0] - INFO [Thread-26:QuorumPeer@1270] - quorum.auth.learnerRequireSasl set to false
2017-01-24 22:12:24,904 [myid:0] - INFO [Thread-26:QuorumPeer@1286] - quorum.auth.kerberos.servicePrincipal set to zkquorum/localhost
2017-01-24 22:12:24,904 [myid:0] - INFO [Thread-26:QuorumPeer@1298] - quorum.auth.server.saslLoginContext set to QuorumServer
2017-01-24 22:12:24,904 [myid:0] - INFO [Thread-26:QuorumPeer@1292] - quorum.auth.learner.saslLoginContext set to QuorumLearner
2017-01-24 22:12:24,904 [myid:0] - INFO [Thread-26:QuorumPeer@1306] - quorum.cnxn.threads.size set to 20
2017-01-24 22:12:24,905 [myid:0] - INFO [Thread-26:Login@294] - QuorumServer successfully logged in.
2017-01-24 22:12:24,905 [myid:0] - INFO [Thread-26:Login@294] - QuorumLearner successfully logged in.
2017-01-24 22:12:24,905 [myid:0] - INFO [Thread-26:QuorumPeer@540] - currentEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
2017-01-24 22:12:24,906 [myid:0] - INFO [Thread-26:QuorumPeer@555] - acceptedEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
2017-01-24 22:12:24,911 [myid:0] - INFO [Thread-27:QuorumCnxManager$Listener@691] - My election bind port: 0.0.0.0/0.0.0.0:11229
2017-01-24 22:12:24,916 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11227:QuorumPeer@781] - LOOKING
2017-01-24 22:12:24,916 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11227:FastLeaderElection@744] - New election. My id = 0, proposed zxid=0x0
2017-01-24 22:12:24,917 [myid:0] - INFO [WorkerReceiver[myid=0]:FastLeaderElection@542] - Notification: 0 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 0 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)
2017-01-24 22:12:24,917 [myid:1] - INFO [localhost/127.0.0.1:11232:QuorumCnxManager$Listener@698] - Received connection request /127.0.0.1:45181
2017-01-24 22:12:24,919 [myid:2] - INFO [localhost/127.0.0.1:11235:QuorumCnxManager$Listener@698] - Received connection request /127.0.0.1:50340
2017-01-24 22:12:24,919 [myid:0] - INFO [localhost/127.0.0.1:11229:QuorumCnxManager$Listener@698] - Received connection request /127.0.0.1:51836
2017-01-24 22:12:24,920 [myid:0] - INFO [QuorumConnectionThread-[myid=0]-1:SaslQuorumAuthLearner@79] - Skipping SASL authentication as quorum.auth.learnerRequireSasl=false
2017-01-24 22:12:24,921 [myid:0] - INFO [QuorumConnectionThread-[myid=0]-2:SaslQuorumAuthLearner@79] - Skipping SASL authentication as quorum.auth.learnerRequireSasl=false
2017-01-24 22:12:24,921 [myid:0] - INFO [QuorumConnectionThread-[myid=0]-2:QuorumCnxManager@331] - Have smaller server identifier, so dropping the connection: (2, 0)
2017-01-24 22:12:24,922 [myid:0] - INFO [QuorumConnectionThread-[myid=0]-1:QuorumCnxManager@331] - Have smaller server identifier, so dropping the connection: (1, 0)
2017-01-24 22:12:24,926 [myid:0] - INFO [localhost/127.0.0.1:11229:QuorumCnxManager$Listener@698] - Received connection request /127.0.0.1:51837
2017-01-24 22:12:24,933 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@542] - Notification: 0 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 0 (n.sid), 0x0 (n.peerEPoch), FOLLOWING (my state)
2017-01-24 22:12:24,933 [myid:0] - INFO [WorkerReceiver[myid=0]:FastLeaderElection@542] - Notification: 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)
2017-01-24 22:12:24,934 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@542] - Notification: 0 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 0 (n.sid), 0x0 (n.peerEPoch), LEADING (my state)
2017-01-24 22:12:24,934 [myid:0] - INFO [WorkerReceiver[myid=0]:FastLeaderElection@542] - Notification: 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 0 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)
2017-01-24 22:12:24,934 [myid:0] - INFO [WorkerReceiver[myid=0]:FastLeaderElection@542] - Notification: 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)
2017-01-24 22:12:24,934 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@542] - Notification: 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 0 (n.sid), 0x0 (n.peerEPoch), LEADING (my state)
2017-01-24 22:12:24,935 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@542] - Notification: 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 0 (n.sid), 0x0 (n.peerEPoch), FOLLOWING (my state)
2017-01-24 22:12:24,937 [myid:0] - INFO [WorkerReceiver[myid=0]:FastLeaderElection@542] - Notification: 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), FOLLOWING (n.state), 1 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)
2017-01-24 22:12:24,937 [myid:0] - INFO [WorkerReceiver[myid=0]:FastLeaderElection@542] - Notification: 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), FOLLOWING (n.state), 1 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)
2017-01-24 22:12:24,937 [myid:0] - INFO [WorkerReceiver[myid=0]:FastLeaderElection@542] - Notification: 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LEADING (n.state), 2 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)
2017-01-24 22:12:24,937 [myid:0] - INFO [WorkerReceiver[myid=0]:FastLeaderElection@542] - Notification: 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LEADING (n.state), 2 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)
2017-01-24 22:12:25,138 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11227:QuorumPeer@849] - FOLLOWING
2017-01-24 22:12:25,138 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11227:ZooKeeperServer@162] - Created server with tickTime 4000 minSessionTimeout 8000 maxSessionTimeout 80000 datadir /data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test6261823493417515862.junit.dir/data/version-2 snapdir /data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test6261823493417515862.junit.dir/data/version-2
2017-01-24 22:12:25,138 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11227:Follower@64] - FOLLOWING - LEADER ELECTION TOOK - 222
2017-01-24 22:12:25,139 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11227:SaslQuorumAuthLearner@79] - Skipping SASL authentication as quorum.auth.learnerRequireSasl=false
2017-01-24 22:12:25,139 [myid:2] - INFO [LearnerHandler-/127.0.0.1:42564:LearnerHandler@287] - Follower sid: 0 : info : org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@559c4c8e
2017-01-24 22:12:25,141 [myid:2] - INFO [LearnerHandler-/127.0.0.1:42564:LearnerHandler@342] - Synchronizing with Follower sid: 0 maxCommittedLog=0x100000002 minCommittedLog=0x100000001 peerLastZxid=0x0
2017-01-24 22:12:25,141 [myid:2] - WARN [LearnerHandler-/127.0.0.1:42564:LearnerHandler@405] - Unhandled proposal scenario
2017-01-24 22:12:25,141 [myid:2] - INFO [LearnerHandler-/127.0.0.1:42564:LearnerHandler@441] - Sending snapshot last zxid of peer is 0x0 zxid of leader is 0x100000002sent zxid of db as 0x100000002
2017-01-24 22:12:25,141 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11227:Learner@329] - Getting a snapshot from leader
2017-01-24 22:12:25,143 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11227:FileTxnSnapLog@281] - Snapshotting: 0x100000002 to /data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test6261823493417515862.junit.dir/data/version-2/snapshot.100000002
2017-01-24 22:12:25,145 [myid:2] - INFO [LearnerHandler-/127.0.0.1:42564:LearnerHandler@477] - Received NEWLEADER-ACK message from 0
2017-01-24 22:12:25,152 [myid:] - INFO [Thread-12:FourLetterWordMain@43] - connecting to 127.0.0.1 11227
2017-01-24 22:12:25,153 [myid:0] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11227:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:42825
2017-01-24 22:12:25,153 [myid:0] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11227:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:42825
2017-01-24 22:12:25,153 [myid:0] - INFO [Thread-29:NIOServerCnxn$StatCommand@655] - Stat command output
2017-01-24 22:12:25,156 [myid:0] - INFO [Thread-29:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:42825 (no session established for client)
2017-01-24 22:12:25,159 [myid:0] - WARN [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11227:Follower@119] - Got zxid 0x100000003 expected 0x1
2017-01-24 22:12:25,159 [myid:0] - INFO [SyncThread:0:FileTxnLog@199] - Creating new log file: log.100000003
2017-01-24 22:12:25,160 [myid:] - INFO [Thread-12:QuorumAuthUpgradeTest@229] - Restarting server myid=1
2017-01-24 22:12:25,161 [myid:] - INFO [Thread-12:QuorumBase@314] - Shutting down quorum peer QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11233
2017-01-24 22:12:25,161 [myid:] - INFO [Thread-12:Leader@491] - Shutting down
2017-01-24 22:12:25,161 [myid:] - INFO [Thread-12:Leader@497] - Shutdown called
java.lang.Exception: shutdown Leader! reason: quorum Peer shutdown
at org.apache.zookeeper.server.quorum.Leader.shutdown(Leader.java:497)
at org.apache.zookeeper.server.quorum.QuorumPeer.shutdown(QuorumPeer.java:893)
at org.apache.zookeeper.test.QuorumBase.shutdown(QuorumBase.java:315)
at org.apache.zookeeper.server.quorum.QuorumPeerTestBase$TestQPMain.shutdown(QuorumPeerTestBase.java:59)
at org.apache.zookeeper.server.quorum.QuorumPeerTestBase$MainThread.shutdown(QuorumPeerTestBase.java:152)
at org.apache.zookeeper.server.quorum.auth.QuorumAuthTestBase.shutdown(QuorumAuthTestBase.java:138)
at org.apache.zookeeper.server.quorum.auth.QuorumAuthUpgradeTest.restartServer(QuorumAuthUpgradeTest.java:230)
at org.apache.zookeeper.server.quorum.auth.QuorumAuthUpgradeTest.testRollingUpgrade(QuorumAuthUpgradeTest.java:195)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:52)
at org.junit.internal.runners.statements.FailOnTimeout$1.run(FailOnTimeout.java:28)
2017-01-24 22:12:25,161 [myid:] - INFO [Thread-12:ZooKeeperServer@419] - shutting down
2017-01-24 22:12:25,161 [myid:2] - INFO [Thread-20:Leader$LearnerCnxAcceptor@318] - exception while shutting down acceptor: java.net.SocketException: Socket closed
2017-01-24 22:12:25,162 [myid:] - INFO [Thread-12:SessionTrackerImpl@225] - Shutting down
2017-01-24 22:12:25,162 [myid:] - INFO [Thread-12:PrepRequestProcessor@761] - Shutting down
2017-01-24 22:12:25,163 [myid:] - INFO [Thread-12:ProposalRequestProcessor@88] - Shutting down
2017-01-24 22:12:25,163 [myid:] - INFO [Thread-12:CommitProcessor@181] - Shutting down
2017-01-24 22:12:25,163 [myid:2] - INFO [ProcessThread(sid:2 cport:-1)::PrepRequestProcessor@143] - PrepRequestProcessor exited loop!
2017-01-24 22:12:25,163 [myid:] - INFO [Thread-12:Leader$ToBeAppliedRequestProcessor@656] - Shutting down
2017-01-24 22:12:25,163 [myid:] - INFO [Thread-12:FinalRequestProcessor@415] - shutdown of request processor complete
2017-01-24 22:12:25,163 [myid:2] - INFO [CommitProcessor:2:CommitProcessor@150] - CommitProcessor exited loop!
2017-01-24 22:12:25,163 [myid:] - INFO [Thread-12:SyncRequestProcessor@175] - Shutting down
2017-01-24 22:12:25,164 [myid:2] - INFO [SyncThread:2:SyncRequestProcessor@155] - SyncRequestProcessor exited!
2017-01-24 22:12:25,164 [myid:0] - WARN [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11227:Follower@90] - Exception when following the leader
java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:392)
at org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63)
at org.apache.zookeeper.server.quorum.QuorumPacket.deserialize(QuorumPacket.java:83)
at org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:99)
at org.apache.zookeeper.server.quorum.Learner.readPacket(Learner.java:152)
at org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:86)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:851)
2017-01-24 22:12:25,164 [myid:1] - WARN [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11230:Follower@90] - Exception when following the leader
java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:392)
at org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63)
at org.apache.zookeeper.server.quorum.QuorumPacket.deserialize(QuorumPacket.java:83)
at org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:99)
at org.apache.zookeeper.server.quorum.Learner.readPacket(Learner.java:152)
at org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:86)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:851)
2017-01-24 22:12:25,165 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11230:Follower@167] - shutdown called
java.lang.Exception: shutdown Follower
at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:167)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:855)
2017-01-24 22:12:25,165 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11230:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:52319 which had sessionid 0x159d440f0ed0000
2017-01-24 22:12:25,165 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11230:FollowerZooKeeperServer@139] - Shutting down
2017-01-24 22:12:25,166 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11230:ZooKeeperServer@419] - shutting down
2017-01-24 22:12:25,166 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11230:FollowerRequestProcessor@105] - Shutting down
2017-01-24 22:12:25,166 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11230:CommitProcessor@181] - Shutting down
2017-01-24 22:12:25,166 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11230:FinalRequestProcessor@415] - shutdown of request processor complete
2017-01-24 22:12:25,166 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11230:SyncRequestProcessor@175] - Shutting down
2017-01-24 22:12:25,166 [myid:1] - INFO [FollowerRequestProcessor:1:FollowerRequestProcessor@95] - FollowerRequestProcessor exited loop!
2017-01-24 22:12:25,164 [myid:2] - WARN [LearnerHandler-/127.0.0.1:42564:LearnerHandler@598] - ******* GOODBYE /127.0.0.1:42564 ********
2017-01-24 22:12:25,166 [myid:1] - INFO [SyncThread:1:SyncRequestProcessor@155] - SyncRequestProcessor exited!
2017-01-24 22:12:25,166 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@224] - NIOServerCnxn factory exited run method
2017-01-24 22:12:25,166 [myid:2] - WARN [LearnerHandler-/127.0.0.1:42553:LearnerHandler@598] - ******* GOODBYE /127.0.0.1:42553 ********
2017-01-24 22:12:25,166 [myid:] - INFO [Thread-12-SendThread(localhost:11230):ClientCnxn$SendThread@1098] - Unable to read additional data from server sessionid 0x159d440f0ed0000, likely server has closed socket, closing socket connection and attempting reconnect
2017-01-24 22:12:25,167 [myid:2] - WARN [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11233:QuorumPeer@867] - Unexpected exception
java.lang.InterruptedException: sleep interrupted
at java.lang.Thread.sleep(Native Method)
at org.apache.zookeeper.server.quorum.Leader.lead(Leader.java:451)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:864)
2017-01-24 22:12:25,167 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11233:Leader@491] - Shutting down
2017-01-24 22:12:25,168 [myid:2] - WARN [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11233:QuorumPeer@879] - QuorumPeer main thread exited
2017-01-24 22:12:25,165 [myid:2] - WARN [LearnerHandler-/127.0.0.1:42554:LearnerHandler@598] - ******* GOODBYE /127.0.0.1:42554 ********
2017-01-24 22:12:25,165 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11227:Follower@167] - shutdown called
java.lang.Exception: shutdown Follower
at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:167)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:855)
2017-01-24 22:12:25,168 [myid:2] - WARN [LearnerHandler-/127.0.0.1:42554:LearnerHandler@610] - Ignoring unexpected exception
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1219)
at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:340)
at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:338)
at org.apache.zookeeper.server.quorum.LearnerHandler.shutdown(LearnerHandler.java:608)
at org.apache.zookeeper.server.quorum.LearnerHandler.run(LearnerHandler.java:601)
2017-01-24 22:12:25,167 [myid:2] - WARN [LearnerHandler-/127.0.0.1:42553:LearnerHandler@610] - Ignoring unexpected exception
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1219)
at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:340)
at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:338)
at org.apache.zookeeper.server.quorum.LearnerHandler.shutdown(LearnerHandler.java:608)
at org.apache.zookeeper.server.quorum.LearnerHandler.run(LearnerHandler.java:601)
2017-01-24 22:12:25,167 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11230:QuorumPeer@781] - LOOKING
2017-01-24 22:12:25,171 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11230:FileSnap@83] - Reading snapshot /data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test8525006967275020093.junit.dir/data/version-2/snapshot.0
2017-01-24 22:12:25,173 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11230:FastLeaderElection@744] - New election. My id = 1, proposed zxid=0x100000003
2017-01-24 22:12:25,174 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@542] - Notification: 1 (n.leader), 0x100000003 (n.zxid), 0x2 (n.round), LOOKING (n.state), 1 (n.sid), 0x1 (n.peerEPoch), LOOKING (my state)
2017-01-24 22:12:25,167 [myid:1] - INFO [CommitProcessor:1:CommitProcessor@150] - CommitProcessor exited loop!
2017-01-24 22:12:25,174 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@542] - Notification: 1 (n.leader), 0x100000003 (n.zxid), 0x2 (n.round), LOOKING (n.state), 1 (n.sid), 0x1 (n.peerEPoch), LOOKING (my state)
2017-01-24 22:12:25,167 [myid:2] - WARN [LearnerHandler-/127.0.0.1:42564:LearnerHandler@610] - Ignoring unexpected exception
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1219)
at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:340)
at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:338)
at org.apache.zookeeper.server.quorum.LearnerHandler.shutdown(LearnerHandler.java:608)
at org.apache.zookeeper.server.quorum.LearnerHandler.run(LearnerHandler.java:601)
2017-01-24 22:12:25,174 [myid:2] - WARN [RecvWorker:1:QuorumCnxManager$RecvWorker@983] - Interrupting SendWorker
2017-01-24 22:12:25,175 [myid:2] - WARN [SendWorker:1:QuorumCnxManager$SendWorker@896] - Interrupted while waiting for message on queue
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1049)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:73)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:884)
2017-01-24 22:12:25,176 [myid:2] - WARN [SendWorker:1:QuorumCnxManager$SendWorker@906] - Send worker leaving thread
2017-01-24 22:12:25,174 [myid:0] - INFO [WorkerReceiver[myid=0]:FastLeaderElection@542] - Notification: 1 (n.leader), 0x100000003 (n.zxid), 0x2 (n.round), LOOKING (n.state), 1 (n.sid), 0x1 (n.peerEPoch), FOLLOWING (my state)
2017-01-24 22:12:25,169 [myid:2] - ERROR [localhost/127.0.0.1:11235:QuorumCnxManager$Listener@715] - Exception while listening
java.net.SocketException: Socket closed
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
at java.net.ServerSocket.implAccept(ServerSocket.java:530)
at java.net.ServerSocket.accept(ServerSocket.java:498)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$Listener.run(QuorumCnxManager.java:696)
2017-01-24 22:12:25,169 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11227:FollowerZooKeeperServer@139] - Shutting down
2017-01-24 22:12:25,176 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11227:ZooKeeperServer@419] - shutting down
2017-01-24 22:12:25,177 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11227:FollowerRequestProcessor@105] - Shutting down
2017-01-24 22:12:25,177 [myid:2] - WARN [RecvWorker:0:QuorumCnxManager$RecvWorker@980] - Connection broken for id 0, my id = 2, error =
java.net.SocketException: Socket closed
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at java.io.BufferedInputStream.read(BufferedInputStream.java:254)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:965)
2017-01-24 22:12:25,177 [myid:2] - WARN [RecvWorker:0:QuorumCnxManager$RecvWorker@983] - Interrupting SendWorker
2017-01-24 22:12:25,175 [myid:0] - WARN [RecvWorker:2:QuorumCnxManager$RecvWorker@980] - Connection broken for id 2, my id = 0, error =
java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:392)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:965)
2017-01-24 22:12:25,177 [myid:0] - WARN [RecvWorker:2:QuorumCnxManager$RecvWorker@983] - Interrupting SendWorker
2017-01-24 22:12:25,175 [myid:1] - WARN [RecvWorker:2:QuorumCnxManager$RecvWorker@980] - Connection broken for id 2, my id = 1, error =
java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:392)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:965)
2017-01-24 22:12:25,178 [myid:1] - WARN [RecvWorker:2:QuorumCnxManager$RecvWorker@983] - Interrupting SendWorker
2017-01-24 22:12:25,178 [myid:] - INFO [Thread-12:QuorumBase@318] - Shutting down leader election QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11233
2017-01-24 22:12:25,178 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@542] - Notification: 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), FOLLOWING (n.state), 0 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)
2017-01-24 22:12:25,178 [myid:2] - WARN [SendWorker:0:QuorumCnxManager$SendWorker@896] - Interrupted while waiting for message on queue
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1049)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:73)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:884)
2017-01-24 22:12:25,179 [myid:2] - WARN [SendWorker:0:QuorumCnxManager$SendWorker@906] - Send worker leaving thread
2017-01-24 22:12:25,179 [myid:] - INFO [Thread-12:QuorumBase@323] - Waiting for QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11233 to exit thread
2017-01-24 22:12:25,180 [myid:] - INFO [Thread-12:QuorumPeerTestBase$MainThread@81] - id = 2 tmpDir = /data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test4813485841779601602.junit.dir clientPort = 11233
2017-01-24 22:12:25,179 [myid:0] - INFO [FollowerRequestProcessor:0:FollowerRequestProcessor@95] - FollowerRequestProcessor exited loop!
2017-01-24 22:12:25,179 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11227:CommitProcessor@181] - Shutting down
2017-01-24 22:12:25,178 [myid:0] - WARN [SendWorker:2:QuorumCnxManager$SendWorker@896] - Interrupted while waiting for message on queue
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1049)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:73)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:884)
2017-01-24 22:12:25,183 [myid:0] - WARN [SendWorker:2:QuorumCnxManager$SendWorker@906] - Send worker leaving thread
2017-01-24 22:12:25,178 [myid:1] - WARN [SendWorker:2:QuorumCnxManager$SendWorker@896] - Interrupted while waiting for message on queue
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1049)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:73)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:884)
2017-01-24 22:12:25,183 [myid:1] - WARN [SendWorker:2:QuorumCnxManager$SendWorker@906] - Send worker leaving thread
2017-01-24 22:12:25,183 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11227:FinalRequestProcessor@415] - shutdown of request processor complete
2017-01-24 22:12:25,183 [myid:0] - INFO [CommitProcessor:0:CommitProcessor@150] - CommitProcessor exited loop!
2017-01-24 22:12:25,182 [myid:] - INFO [Thread-30:QuorumPeerConfig@111] - Reading configuration from: /data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test4813485841779601602.junit.dir/zoo.cfg
2017-01-24 22:12:25,182 [myid:] - INFO [Thread-12:FourLetterWordMain@43] - connecting to 127.0.0.1 11233
2017-01-24 22:12:25,183 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11227:SyncRequestProcessor@175] - Shutting down
2017-01-24 22:12:25,184 [myid:0] - INFO [SyncThread:0:SyncRequestProcessor@155] - SyncRequestProcessor exited!
2017-01-24 22:12:25,185 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11227:QuorumPeer@781] - LOOKING
2017-01-24 22:12:25,185 [myid:] - INFO [Thread-30:QuorumPeerConfig@374] - Defaulting to majority quorums
2017-01-24 22:12:25,186 [myid:2] - INFO [Thread-30:DatadirCleanupManager@78] - autopurge.snapRetainCount set to 3
2017-01-24 22:12:25,186 [myid:2] - INFO [Thread-30:DatadirCleanupManager@79] - autopurge.purgeInterval set to 0
2017-01-24 22:12:25,186 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11227:FileSnap@83] - Reading snapshot /data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test6261823493417515862.junit.dir/data/version-2/snapshot.100000002
2017-01-24 22:12:25,186 [myid:2] - INFO [Thread-30:DatadirCleanupManager@101] - Purge task is not scheduled.
2017-01-24 22:12:25,186 [myid:2] - WARN [Thread-30:QuorumPeerMain@129] - Unable to register log4j JMX control
javax.management.InstanceAlreadyExistsException: log4j:hiearchy=default
at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:437)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1898)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:966)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324)
at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
at org.apache.zookeeper.jmx.ManagedUtil.registerLog4jMBeans(ManagedUtil.java:53)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:127)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:116)
at org.apache.zookeeper.server.quorum.QuorumPeerTestBase$MainThread.run(QuorumPeerTestBase.java:140)
at java.lang.Thread.run(Thread.java:745)
2017-01-24 22:12:25,187 [myid:2] - INFO [Thread-30:QuorumPeerMain@132] - Starting quorum peer
2017-01-24 22:12:25,187 [myid:2] - INFO [Thread-30:NIOServerCnxnFactory@94] - binding to port 0.0.0.0/0.0.0.0:11233
2017-01-24 22:12:25,188 [myid:] - INFO [Thread-12:ClientBase@246] - server 127.0.0.1:11233 not up java.net.ConnectException: Connection refused
2017-01-24 22:12:25,187 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11227:FastLeaderElection@744] - New election. My id = 0, proposed zxid=0x100000003
2017-01-24 22:12:25,188 [myid:0] - INFO [WorkerReceiver[myid=0]:FastLeaderElection@542] - Notification: 0 (n.leader), 0x100000003 (n.zxid), 0x2 (n.round), LOOKING (n.state), 0 (n.sid), 0x1 (n.peerEPoch), LOOKING (my state)
2017-01-24 22:12:25,188 [myid:0] - WARN [WorkerSender[myid=0]:QuorumCnxManager@559] - Cannot open channel to 2 at election address localhost/127.0.0.1:11235
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:538)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:514)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:393)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:365)
at java.lang.Thread.run(Thread.java:745)
2017-01-24 22:12:25,189 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@542] - Notification: 0 (n.leader), 0x100000003 (n.zxid), 0x2 (n.round), LOOKING (n.state), 0 (n.sid), 0x1 (n.peerEPoch), LOOKING (my state)
2017-01-24 22:12:25,189 [myid:2] - INFO [Thread-30:QuorumPeer@1048] - minSessionTimeout set to -1
2017-01-24 22:12:25,189 [myid:2] - INFO [Thread-30:QuorumPeer@1059] - maxSessionTimeout set to -1
2017-01-24 22:12:25,189 [myid:2] - INFO [Thread-30:QuorumPeer@1279] - quorum.auth.enableSasl set to true
2017-01-24 22:12:25,189 [myid:2] - INFO [Thread-30:QuorumPeer@1264] - quorum.auth.serverRequireSasl set to false
2017-01-24 22:12:25,189 [myid:2] - INFO [Thread-30:QuorumPeer@1270] - quorum.auth.learnerRequireSasl set to false
2017-01-24 22:12:25,190 [myid:2] - INFO [Thread-30:QuorumPeer@1286] - quorum.auth.kerberos.servicePrincipal set to zkquorum/localhost
2017-01-24 22:12:25,190 [myid:2] - INFO [Thread-30:QuorumPeer@1298] - quorum.auth.server.saslLoginContext set to QuorumServer
2017-01-24 22:12:25,190 [myid:2] - INFO [Thread-30:QuorumPeer@1292] - quorum.auth.learner.saslLoginContext set to QuorumLearner
2017-01-24 22:12:25,190 [myid:2] - INFO [Thread-30:QuorumPeer@1306] - quorum.cnxn.threads.size set to 20
2017-01-24 22:12:25,190 [myid:2] - INFO [Thread-30:Login@294] - QuorumServer successfully logged in.
2017-01-24 22:12:25,190 [myid:2] - INFO [Thread-30:Login@294] - QuorumLearner successfully logged in.
2017-01-24 22:12:25,191 [myid:2] - INFO [Thread-30:QuorumPeer@540] - currentEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
2017-01-24 22:12:25,192 [myid:2] - INFO [Thread-30:QuorumPeer@555] - acceptedEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
2017-01-24 22:12:25,193 [myid:2] - INFO [Thread-31:QuorumCnxManager$Listener@691] - My election bind port: 0.0.0.0/0.0.0.0:11235
2017-01-24 22:12:25,196 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11233:QuorumPeer@781] - LOOKING
2017-01-24 22:12:25,196 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11233:FastLeaderElection@744] - New election. My id = 2, proposed zxid=0x0
2017-01-24 22:12:25,197 [myid:0] - INFO [localhost/127.0.0.1:11229:QuorumCnxManager$Listener@698] - Received connection request /127.0.0.1:51842
2017-01-24 22:12:25,197 [myid:1] - INFO [localhost/127.0.0.1:11232:QuorumCnxManager$Listener@698] - Received connection request /127.0.0.1:45190
2017-01-24 22:12:25,198 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@542] - Notification: 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)
2017-01-24 22:12:25,202 [myid:2] - INFO [QuorumConnectionThread-[myid=2]-2:SaslQuorumAuthLearner@79] - Skipping SASL authentication as quorum.auth.learnerRequireSasl=false
2017-01-24 22:12:25,204 [myid:2] - INFO [QuorumConnectionThread-[myid=2]-1:SaslQuorumAuthLearner@79] - Skipping SASL authentication as quorum.auth.learnerRequireSasl=false
2017-01-24 22:12:25,210 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@542] - Notification: 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)
2017-01-24 22:12:25,210 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@542] - Notification: 1 (n.leader), 0x100000003 (n.zxid), 0x2 (n.round), LOOKING (n.state), 1 (n.sid), 0x1 (n.peerEPoch), LOOKING (my state)
2017-01-24 22:12:25,210 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@542] - Notification: 1 (n.leader), 0x100000003 (n.zxid), 0x2 (n.round), LOOKING (n.state), 1 (n.sid), 0x1 (n.peerEPoch), LOOKING (my state)
2017-01-24 22:12:25,212 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@542] - Notification: 0 (n.leader), 0x100000003 (n.zxid), 0x2 (n.round), LOOKING (n.state), 0 (n.sid), 0x1 (n.peerEPoch), LOOKING (my state)
2017-01-24 22:12:25,212 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@542] - Notification: 1 (n.leader), 0x100000003 (n.zxid), 0x2 (n.round), LOOKING (n.state), 2 (n.sid), 0x1 (n.peerEPoch), LOOKING (my state)
2017-01-24 22:12:25,212 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@542] - Notification: 1 (n.leader), 0x100000003 (n.zxid), 0x2 (n.round), LOOKING (n.state), 2 (n.sid), 0x1 (n.peerEPoch), LOOKING (my state)
2017-01-24 22:12:25,213 [myid:0] - INFO [WorkerReceiver[myid=0]:FastLeaderElection@542] - Notification: 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)
2017-01-24 22:12:25,213 [myid:0] - INFO [WorkerReceiver[myid=0]:FastLeaderElection@542] - Notification: 1 (n.leader), 0x100000003 (n.zxid), 0x2 (n.round), LOOKING (n.state), 2 (n.sid), 0x1 (n.peerEPoch), LOOKING (my state)
2017-01-24 22:12:25,214 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@542] - Notification: 1 (n.leader), 0x100000003 (n.zxid), 0x2 (n.round), LOOKING (n.state), 0 (n.sid), 0x1 (n.peerEPoch), LOOKING (my state)
2017-01-24 22:12:25,214 [myid:0] - INFO [WorkerReceiver[myid=0]:FastLeaderElection@542] - Notification: 1 (n.leader), 0x100000003 (n.zxid), 0x2 (n.round), LOOKING (n.state), 0 (n.sid), 0x1 (n.peerEPoch), LOOKING (my state)
2017-01-24 22:12:25,214 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@542] - Notification: 0 (n.leader), 0x100000003 (n.zxid), 0x2 (n.round), LOOKING (n.state), 0 (n.sid), 0x1 (n.peerEPoch), LOOKING (my state)
2017-01-24 22:12:25,214 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@542] - Notification: 1 (n.leader), 0x100000003 (n.zxid), 0x2 (n.round), LOOKING (n.state), 0 (n.sid), 0x1 (n.peerEPoch), LOOKING (my state)
2017-01-24 22:12:25,414 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11230:QuorumPeer@861] - LEADING
2017-01-24 22:12:25,414 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11230:ZooKeeperServer@162] - Created server with tickTime 4000 minSessionTimeout 8000 maxSessionTimeout 80000 datadir /data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test8525006967275020093.junit.dir/data/version-2 snapdir /data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test8525006967275020093.junit.dir/data/version-2
2017-01-24 22:12:25,415 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11230:Leader@356] - LEADING - LEADER ELECTION TOOK - 245
2017-01-24 22:12:25,415 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11230:FileSnap@83] - Reading snapshot /data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test8525006967275020093.junit.dir/data/version-2/snapshot.0
2017-01-24 22:12:25,416 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11227:QuorumPeer@849] - FOLLOWING
2017-01-24 22:12:25,416 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11227:ZooKeeperServer@162] - Created server with tickTime 4000 minSessionTimeout 8000 maxSessionTimeout 80000 datadir /data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test6261823493417515862.junit.dir/data/version-2 snapdir /data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test6261823493417515862.junit.dir/data/version-2
2017-01-24 22:12:25,416 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11227:Follower@64] - FOLLOWING - LEADER ELECTION TOOK - 231
2017-01-24 22:12:25,417 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11227:SaslQuorumAuthLearner@79] - Skipping SASL authentication as quorum.auth.learnerRequireSasl=false
2017-01-24 22:12:25,417 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11230:FileTxnSnapLog@281] - Snapshotting: 0x100000003 to /data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test8525006967275020093.junit.dir/data/version-2/snapshot.100000003
2017-01-24 22:12:25,415 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11233:QuorumPeer@849] - FOLLOWING
2017-01-24 22:12:25,417 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11233:ZooKeeperServer@162] - Created server with tickTime 4000 minSessionTimeout 8000 maxSessionTimeout 80000 datadir /data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test4813485841779601602.junit.dir/data/version-2 snapdir /data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test4813485841779601602.junit.dir/data/version-2
2017-01-24 22:12:25,417 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11233:Follower@64] - FOLLOWING - LEADER ELECTION TOOK - 221
2017-01-24 22:12:25,418 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11233:SaslQuorumAuthLearner@79] - Skipping SASL authentication as quorum.auth.learnerRequireSasl=false
2017-01-24 22:12:25,422 [myid:1] - INFO [LearnerHandler-/127.0.0.1:50169:LearnerHandler@287] - Follower sid: 0 : info : org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@752a0b2f
2017-01-24 22:12:25,422 [myid:1] - INFO [LearnerHandler-/127.0.0.1:50170:LearnerHandler@287] - Follower sid: 2 : info : org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@1b4476c1
2017-01-24 22:12:25,424 [myid:1] - INFO [LearnerHandler-/127.0.0.1:50169:LearnerHandler@342] - Synchronizing with Follower sid: 0 maxCommittedLog=0x0 minCommittedLog=0x0 peerLastZxid=0x100000003
2017-01-24 22:12:25,424 [myid:1] - INFO [LearnerHandler-/127.0.0.1:50169:LearnerHandler@441] - Sending snapshot last zxid of peer is 0x100000003 zxid of leader is 0x200000000sent zxid of db as 0x100000003
2017-01-24 22:12:25,424 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11227:Learner@329] - Getting a snapshot from leader
2017-01-24 22:12:25,425 [myid:1] - INFO [LearnerHandler-/127.0.0.1:50170:LearnerHandler@342] - Synchronizing with Follower sid: 2 maxCommittedLog=0x0 minCommittedLog=0x0 peerLastZxid=0x0
2017-01-24 22:12:25,426 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11233:Learner@329] - Getting a snapshot from leader
2017-01-24 22:12:25,426 [myid:1] - INFO [LearnerHandler-/127.0.0.1:50170:LearnerHandler@441] - Sending snapshot last zxid of peer is 0x0 zxid of leader is 0x200000000sent zxid of db as 0x100000003
2017-01-24 22:12:25,429 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11233:FileTxnSnapLog@281] - Snapshotting: 0x100000003 to /data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test4813485841779601602.junit.dir/data/version-2/snapshot.100000003
2017-01-24 22:12:25,429 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11227:FileTxnSnapLog@281] - Snapshotting: 0x100000003 to /data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test6261823493417515862.junit.dir/data/version-2/snapshot.100000003
2017-01-24 22:12:25,432 [myid:1] - INFO [LearnerHandler-/127.0.0.1:50170:LearnerHandler@477] - Received NEWLEADER-ACK message from 2
2017-01-24 22:12:25,432 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11230:Leader@934] - Have quorum of supporters, sids: [ 1,2 ]; starting up and setting last processed zxid: 0x200000000
2017-01-24 22:12:25,433 [myid:1] - INFO [LearnerHandler-/127.0.0.1:50169:LearnerHandler@477] - Received NEWLEADER-ACK message from 0
2017-01-24 22:12:25,446 [myid:] - INFO [Thread-12:FourLetterWordMain@43] - connecting to 127.0.0.1 11233
2017-01-24 22:12:25,446 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45802
2017-01-24 22:12:25,447 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:45802
2017-01-24 22:12:25,447 [myid:2] - INFO [Thread-35:NIOServerCnxn$StatCommand@655] - Stat command output
2017-01-24 22:12:25,448 [myid:2] - INFO [Thread-35:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45802 (no session established for client)
2017-01-24 22:12:25,686 [myid:0] - INFO [WorkerSender[myid=0]:FastLeaderElection$Messenger$WorkerSender@370] - WorkerSender is down
2017-01-24 22:12:25,687 [myid:0] - INFO [WorkerReceiver[myid=0]:FastLeaderElection$Messenger$WorkerReceiver@340] - WorkerReceiver is down
2017-01-24 22:12:26,176 [myid:2] - INFO [localhost/127.0.0.1:11235:QuorumCnxManager$Listener@728] - Leaving listener
2017-01-24 22:12:26,212 [myid:] - WARN [Thread-12-SendThread(localhost:11233):ClientCnxn$SendThread@957] - SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: '/data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test2999887027410032136.junit.dir/jaas.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.
2017-01-24 22:12:26,212 [myid:] - INFO [Thread-12-SendThread(localhost:11233):ClientCnxn$SendThread@975] - Opening socket connection to server localhost/127.0.0.1:11233
2017-01-24 22:12:26,212 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45803
2017-01-24 22:12:26,212 [myid:] - INFO [Thread-12-SendThread(localhost:11233):ClientCnxn$SendThread@852] - Socket connection established, initiating session, client: /127.0.0.1:45803, server: localhost/127.0.0.1:11233
2017-01-24 22:12:26,213 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:ZooKeeperServer@832] - Client attempting to renew session 0x159d440f0ed0000 at /127.0.0.1:45803
2017-01-24 22:12:26,213 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:Learner@107] - Revalidating client: 0x159d440f0ed0000
2017-01-24 22:12:26,215 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11233:ZooKeeperServer@595] - Established session 0x159d440f0ed0000 with negotiated timeout 30000 for client /127.0.0.1:45803
2017-01-24 22:12:26,215 [myid:] - INFO [Thread-12-SendThread(localhost:11233):ClientCnxn$SendThread@1235] - Session establishment complete on server localhost/127.0.0.1:11233, sessionid = 0x159d440f0ed0000, negotiated timeout = 30000
2017-01-24 22:12:26,216 [myid:2] - WARN [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11233:Follower@119] - Got zxid 0x200000001 expected 0x1
2017-01-24 22:12:26,217 [myid:0] - WARN [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11227:Follower@119] - Got zxid 0x200000001 expected 0x1
2017-01-24 22:12:26,217 [myid:2] - INFO [SyncThread:2:FileTxnLog@199] - Creating new log file: log.200000001
2017-01-24 22:12:26,219 [myid:] - INFO [Thread-12:QuorumAuthUpgradeTest@229] - Restarting server myid=2
2017-01-24 22:12:26,220 [myid:] - INFO [Thread-12:QuorumBase@314] - Shutting down quorum peer QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11233
2017-01-24 22:12:26,220 [myid:] - INFO [Thread-12:Follower@167] - shutdown called
java.lang.Exception: shutdown Follower
at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:167)
at org.apache.zookeeper.server.quorum.QuorumPeer.shutdown(QuorumPeer.java:896)
at org.apache.zookeeper.test.QuorumBase.shutdown(QuorumBase.java:315)
at org.apache.zookeeper.server.quorum.QuorumPeerTestBase$TestQPMain.shutdown(QuorumPeerTestBase.java:59)
at org.apache.zookeeper.server.quorum.QuorumPeerTestBase$MainThread.shutdown(QuorumPeerTestBase.java:152)
at org.apache.zookeeper.server.quorum.auth.QuorumAuthTestBase.shutdown(QuorumAuthTestBase.java:138)
at org.apache.zookeeper.server.quorum.auth.QuorumAuthUpgradeTest.restartServer(QuorumAuthUpgradeTest.java:230)
at org.apache.zookeeper.server.quorum.auth.QuorumAuthUpgradeTest.testRollingUpgrade(QuorumAuthUpgradeTest.java:196)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:52)
at org.junit.internal.runners.statements.FailOnTimeout$1.run(FailOnTimeout.java:28)
2017-01-24 22:12:26,220 [myid:] - INFO [Thread-12:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45803 which had sessionid 0x159d440f0ed0000
2017-01-24 22:12:26,221 [myid:] - INFO [Thread-12-SendThread(localhost:11233):ClientCnxn$SendThread@1098] - Unable to read additional data from server sessionid 0x159d440f0ed0000, likely server has closed socket, closing socket connection and attempting reconnect
2017-01-24 22:12:26,221 [myid:] - INFO [Thread-12:FollowerZooKeeperServer@139] - Shutting down
2017-01-24 22:12:26,221 [myid:] - INFO [Thread-12:ZooKeeperServer@419] - shutting down
2017-01-24 22:12:26,221 [myid:] - INFO [Thread-12:FollowerRequestProcessor@105] - Shutting down
2017-01-24 22:12:26,221 [myid:] - INFO [Thread-12:CommitProcessor@181] - Shutting down
2017-01-24 22:12:26,221 [myid:2] - INFO [FollowerRequestProcessor:2:FollowerRequestProcessor@95] - FollowerRequestProcessor exited loop!
2017-01-24 22:12:26,221 [myid:2] - INFO [CommitProcessor:2:CommitProcessor@150] - CommitProcessor exited loop!
2017-01-24 22:12:26,221 [myid:] - INFO [Thread-12:FinalRequestProcessor@415] - shutdown of request processor complete
2017-01-24 22:12:26,222 [myid:] - INFO [Thread-12:SyncRequestProcessor@175] - Shutting down
2017-01-24 22:12:26,222 [myid:2] - INFO [SyncThread:2:SyncRequestProcessor@155] - SyncRequestProcessor exited!
2017-01-24 22:12:26,223 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@224] - NIOServerCnxn factory exited run method
2017-01-24 22:12:26,223 [myid:2] - ERROR [localhost/127.0.0.1:11235:QuorumCnxManager$Listener@715] - Exception while listening
java.net.SocketException: Socket closed
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
at java.net.ServerSocket.implAccept(ServerSocket.java:530)
at java.net.ServerSocket.accept(ServerSocket.java:498)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$Listener.run(QuorumCnxManager.java:696)
2017-01-24 22:12:26,223 [myid:0] - WARN [RecvWorker:2:QuorumCnxManager$RecvWorker@980] - Connection broken for id 2, my id = 0, error =
java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:392)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:965)
2017-01-24 22:12:26,224 [myid:0] - WARN [RecvWorker:2:QuorumCnxManager$RecvWorker@983] - Interrupting SendWorker
2017-01-24 22:12:26,224 [myid:0] - WARN [SendWorker:2:QuorumCnxManager$SendWorker@896] - Interrupted while waiting for message on queue
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1049)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:73)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:884)
2017-01-24 22:12:26,224 [myid:2] - WARN [RecvWorker:0:QuorumCnxManager$RecvWorker@980] - Connection broken for id 0, my id = 2, error =
java.net.SocketException: Socket closed
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at java.io.BufferedInputStream.read(BufferedInputStream.java:254)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:965)
2017-01-24 22:12:26,225 [myid:2] - WARN [RecvWorker:0:QuorumCnxManager$RecvWorker@983] - Interrupting SendWorker
2017-01-24 22:12:26,225 [myid:2] - WARN [SendWorker:1:QuorumCnxManager$SendWorker@896] - Interrupted while waiting for message on queue
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1049)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:73)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:884)
2017-01-24 22:12:26,226 [myid:2] - WARN [SendWorker:1:QuorumCnxManager$SendWorker@906] - Send worker leaving thread
2017-01-24 22:12:26,225 [myid:0] - WARN [SendWorker:2:QuorumCnxManager$SendWorker@906] - Send worker leaving thread
2017-01-24 22:12:26,224 [myid:2] - WARN [SendWorker:0:QuorumCnxManager$SendWorker@896] - Interrupted while waiting for message on queue
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1049)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:73)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:884)
2017-01-24 22:12:26,226 [myid:2] - WARN [SendWorker:0:QuorumCnxManager$SendWorker@906] - Send worker leaving thread
2017-01-24 22:12:26,225 [myid:1] - WARN [RecvWorker:2:QuorumCnxManager$RecvWorker@980] - Connection broken for id 2, my id = 1, error =
java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:392)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:965)
2017-01-24 22:12:26,227 [myid:1] - WARN [RecvWorker:2:QuorumCnxManager$RecvWorker@983] - Interrupting SendWorker
2017-01-24 22:12:26,225 [myid:2] - WARN [RecvWorker:1:QuorumCnxManager$RecvWorker@980] - Connection broken for id 1, my id = 2, error =
java.net.SocketException: Socket closed
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at java.io.BufferedInputStream.read(BufferedInputStream.java:254)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:965)
2017-01-24 22:12:26,227 [myid:2] - WARN [RecvWorker:1:QuorumCnxManager$RecvWorker@983] - Interrupting SendWorker
2017-01-24 22:12:26,227 [myid:1] - WARN [SendWorker:2:QuorumCnxManager$SendWorker@896] - Interrupted while waiting for message on queue
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1049)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:73)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:884)
2017-01-24 22:12:26,228 [myid:1] - WARN [SendWorker:2:QuorumCnxManager$SendWorker@906] - Send worker leaving thread
2017-01-24 22:12:26,228 [myid:] - INFO [Thread-12:QuorumBase@318] - Shutting down leader election QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11233
2017-01-24 22:12:26,229 [myid:] - INFO [Thread-12:QuorumBase@323] - Waiting for QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11233 to exit thread
2017-01-24 22:12:26,611 [myid:] - WARN [Thread-12-SendThread(localhost:11227):ClientCnxn$SendThread@957] - SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: '/data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test2999887027410032136.junit.dir/jaas.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.
2017-01-24 22:12:26,611 [myid:] - INFO [Thread-12-SendThread(localhost:11227):ClientCnxn$SendThread@975] - Opening socket connection to server localhost/127.0.0.1:11227
2017-01-24 22:12:26,612 [myid:0] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11227:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:42834
2017-01-24 22:12:26,612 [myid:] - INFO [Thread-12-SendThread(localhost:11227):ClientCnxn$SendThread@852] - Socket connection established, initiating session, client: /127.0.0.1:42834, server: localhost/127.0.0.1:11227
2017-01-24 22:12:26,612 [myid:0] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11227:ZooKeeperServer@832] - Client attempting to renew session 0x159d440f0ed0000 at /127.0.0.1:42834
2017-01-24 22:12:26,613 [myid:0] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11227:Learner@107] - Revalidating client: 0x159d440f0ed0000
2017-01-24 22:12:26,614 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11227:ZooKeeperServer@595] - Established session 0x159d440f0ed0000 with negotiated timeout 30000 for client /127.0.0.1:42834
2017-01-24 22:12:26,614 [myid:] - INFO [Thread-12-SendThread(localhost:11227):ClientCnxn$SendThread@1235] - Session establishment complete on server localhost/127.0.0.1:11227, sessionid = 0x159d440f0ed0000, negotiated timeout = 30000
2017-01-24 22:12:27,224 [myid:2] - INFO [localhost/127.0.0.1:11235:QuorumCnxManager$Listener@728] - Leaving listener
2017-01-24 22:12:27,435 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11233:Follower@167] - shutdown called
java.lang.Exception: shutdown Follower
at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:167)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:855)
2017-01-24 22:12:27,435 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11233:FollowerZooKeeperServer@139] - Shutting down
2017-01-24 22:12:27,435 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11233:ZooKeeperServer@419] - shutting down
2017-01-24 22:12:27,435 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11233:FollowerRequestProcessor@105] - Shutting down
2017-01-24 22:12:27,436 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11233:CommitProcessor@181] - Shutting down
2017-01-24 22:12:27,436 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11233:FinalRequestProcessor@415] - shutdown of request processor complete
2017-01-24 22:12:27,436 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11233:SyncRequestProcessor@175] - Shutting down
2017-01-24 22:12:27,436 [myid:2] - WARN [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11233:QuorumPeer@879] - QuorumPeer main thread exited
2017-01-24 22:12:27,447 [myid:] - INFO [Thread-12:QuorumPeerTestBase$MainThread@81] - id = 2 tmpDir = /data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test2397521463065956250.junit.dir clientPort = 11233
2017-01-24 22:12:27,452 [myid:] - INFO [Thread-12:FourLetterWordMain@43] - connecting to 127.0.0.1 11233
2017-01-24 22:12:27,453 [myid:] - INFO [Thread-12:ClientBase@246] - server 127.0.0.1:11233 not up java.net.ConnectException: Connection refused
2017-01-24 22:12:27,453 [myid:] - INFO [Thread-36:QuorumPeerConfig@111] - Reading configuration from: /data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test2397521463065956250.junit.dir/zoo.cfg
2017-01-24 22:12:27,454 [myid:] - INFO [Thread-36:QuorumPeerConfig@374] - Defaulting to majority quorums
2017-01-24 22:12:27,454 [myid:2] - INFO [Thread-36:DatadirCleanupManager@78] - autopurge.snapRetainCount set to 3
2017-01-24 22:12:27,454 [myid:2] - INFO [Thread-36:DatadirCleanupManager@79] - autopurge.purgeInterval set to 0
2017-01-24 22:12:27,454 [myid:2] - INFO [Thread-36:DatadirCleanupManager@101] - Purge task is not scheduled.
2017-01-24 22:12:27,455 [myid:2] - WARN [Thread-36:QuorumPeerMain@129] - Unable to register log4j JMX control
javax.management.InstanceAlreadyExistsException: log4j:hiearchy=default
at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:437)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1898)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:966)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324)
at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
at org.apache.zookeeper.jmx.ManagedUtil.registerLog4jMBeans(ManagedUtil.java:53)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:127)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:116)
at org.apache.zookeeper.server.quorum.QuorumPeerTestBase$MainThread.run(QuorumPeerTestBase.java:140)
at java.lang.Thread.run(Thread.java:745)
2017-01-24 22:12:27,455 [myid:2] - INFO [Thread-36:QuorumPeerMain@132] - Starting quorum peer
2017-01-24 22:12:27,456 [myid:2] - INFO [Thread-36:NIOServerCnxnFactory@94] - binding to port 0.0.0.0/0.0.0.0:11233
2017-01-24 22:12:27,456 [myid:2] - INFO [Thread-36:QuorumPeer@1048] - minSessionTimeout set to -1
2017-01-24 22:12:27,456 [myid:2] - INFO [Thread-36:QuorumPeer@1059] - maxSessionTimeout set to -1
2017-01-24 22:12:27,457 [myid:2] - INFO [Thread-36:QuorumPeer@1279] - quorum.auth.enableSasl set to true
2017-01-24 22:12:27,457 [myid:2] - INFO [Thread-36:QuorumPeer@1264] - quorum.auth.serverRequireSasl set to false
2017-01-24 22:12:27,458 [myid:2] - INFO [Thread-36:QuorumPeer@1270] - quorum.auth.learnerRequireSasl set to false
2017-01-24 22:12:27,458 [myid:2] - INFO [Thread-36:QuorumPeer@1286] - quorum.auth.kerberos.servicePrincipal set to zkquorum/localhost
2017-01-24 22:12:27,458 [myid:2] - INFO [Thread-36:QuorumPeer@1298] - quorum.auth.server.saslLoginContext set to QuorumServer
2017-01-24 22:12:27,459 [myid:2] - INFO [Thread-36:QuorumPeer@1292] - quorum.auth.learner.saslLoginContext set to QuorumLearner
2017-01-24 22:12:27,459 [myid:2] - INFO [Thread-36:QuorumPeer@1306] - quorum.cnxn.threads.size set to 20
2017-01-24 22:12:27,459 [myid:2] - INFO [Thread-36:Login@294] - QuorumServer successfully logged in.
2017-01-24 22:12:27,459 [myid:2] - INFO [Thread-36:Login@294] - QuorumLearner successfully logged in.
2017-01-24 22:12:27,460 [myid:2] - INFO [Thread-36:QuorumPeer@540] - currentEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
2017-01-24 22:12:27,461 [myid:2] - INFO [Thread-36:QuorumPeer@555] - acceptedEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
2017-01-24 22:12:27,479 [myid:2] - INFO [Thread-37:QuorumCnxManager$Listener@691] - My election bind port: 0.0.0.0/0.0.0.0:11235
2017-01-24 22:12:27,481 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11233:QuorumPeer@781] - LOOKING
2017-01-24 22:12:27,481 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11233:FastLeaderElection@744] - New election. My id = 2, proposed zxid=0x0
2017-01-24 22:12:27,482 [myid:0] - INFO [localhost/127.0.0.1:11229:QuorumCnxManager$Listener@698] - Received connection request /127.0.0.1:51850
2017-01-24 22:12:27,483 [myid:1] - INFO [localhost/127.0.0.1:11232:QuorumCnxManager$Listener@698] - Received connection request /127.0.0.1:45198
2017-01-24 22:12:27,483 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@542] - Notification: 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)
2017-01-24 22:12:27,488 [myid:2] - INFO [QuorumConnectionThread-[myid=2]-1:SaslQuorumAuthLearner@79] - Skipping SASL authentication as quorum.auth.learnerRequireSasl=false
2017-01-24 22:12:27,493 [myid:2] - INFO [QuorumConnectionThread-[myid=2]-2:SaslQuorumAuthLearner@79] - Skipping SASL authentication as quorum.auth.learnerRequireSasl=false
2017-01-24 22:12:27,496 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@542] - Notification: 1 (n.leader), 0x100000003 (n.zxid), 0x2 (n.round), LOOKING (n.state), 1 (n.sid), 0x1 (n.peerEPoch), LOOKING (my state)
2017-01-24 22:12:27,497 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@542] - Notification: 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x0 (n.peerEPoch), LEADING (my state)
2017-01-24 22:12:27,501 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@542] - Notification: 1 (n.leader), 0x100000003 (n.zxid), 0x2 (n.round), LEADING (n.state), 1 (n.sid), 0x1 (n.peerEPoch), LOOKING (my state)
2017-01-24 22:12:27,501 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@542] - Notification: 1 (n.leader), 0x100000003 (n.zxid), 0x2 (n.round), LOOKING (n.state), 2 (n.sid), 0x1 (n.peerEPoch), LOOKING (my state)
2017-01-24 22:12:27,502 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@542] - Notification: 1 (n.leader), 0x100000003 (n.zxid), 0x2 (n.round), LOOKING (n.state), 2 (n.sid), 0x1 (n.peerEPoch), LEADING (my state)
2017-01-24 22:12:27,502 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@542] - Notification: 1 (n.leader), 0x100000003 (n.zxid), 0x2 (n.round), LOOKING (n.state), 0 (n.sid), 0x1 (n.peerEPoch), LOOKING (my state)
2017-01-24 22:12:27,502 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@542] - Notification: 1 (n.leader), 0x100000003 (n.zxid), 0x2 (n.round), LEADING (n.state), 1 (n.sid), 0x1 (n.peerEPoch), LOOKING (my state)
2017-01-24 22:12:27,502 [myid:0] - INFO [WorkerReceiver[myid=0]:FastLeaderElection@542] - Notification: 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x0 (n.peerEPoch), FOLLOWING (my state)
2017-01-24 22:12:27,502 [myid:0] - INFO [WorkerReceiver[myid=0]:FastLeaderElection@542] - Notification: 1 (n.leader), 0x100000003 (n.zxid), 0x2 (n.round), LOOKING (n.state), 2 (n.sid), 0x1 (n.peerEPoch), FOLLOWING (my state)
2017-01-24 22:12:27,503 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@542] - Notification: 1 (n.leader), 0x100000003 (n.zxid), 0x2 (n.round), FOLLOWING (n.state), 0 (n.sid), 0x1 (n.peerEPoch), LOOKING (my state)
2017-01-24 22:12:27,503 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@542] - Notification: 1 (n.leader), 0x100000003 (n.zxid), 0x2 (n.round), FOLLOWING (n.state), 0 (n.sid), 0x1 (n.peerEPoch), LOOKING (my state)
2017-01-24 22:12:27,703 [myid:] - INFO [Thread-12:FourLetterWordMain@43] - connecting to 127.0.0.1 11233
2017-01-24 22:12:27,704 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11233:QuorumPeer@849] - FOLLOWING
2017-01-24 22:12:27,704 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11233:ZooKeeperServer@162] - Created server with tickTime 4000 minSessionTimeout 8000 maxSessionTimeout 80000 datadir /data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test2397521463065956250.junit.dir/data/version-2 snapdir /data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test2397521463065956250.junit.dir/data/version-2
2017-01-24 22:12:27,704 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45808
2017-01-24 22:12:27,704 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11233:Follower@64] - FOLLOWING - LEADER ELECTION TOOK - 223
2017-01-24 22:12:27,705 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:45808
2017-01-24 22:12:27,705 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11233:SaslQuorumAuthLearner@79] - Skipping SASL authentication as quorum.auth.learnerRequireSasl=false
2017-01-24 22:12:27,706 [myid:1] - INFO [LearnerHandler-/127.0.0.1:50178:LearnerHandler@287] - Follower sid: 2 : info : org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@1b4476c1
2017-01-24 22:12:27,709 [myid:2] - INFO [Thread-38:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45808 (no session established for client)
2017-01-24 22:12:27,710 [myid:1] - INFO [LearnerHandler-/127.0.0.1:50178:LearnerHandler@342] - Synchronizing with Follower sid: 2 maxCommittedLog=0x200000001 minCommittedLog=0x200000001 peerLastZxid=0x0
2017-01-24 22:12:27,710 [myid:1] - WARN [LearnerHandler-/127.0.0.1:50178:LearnerHandler@405] - Unhandled proposal scenario
2017-01-24 22:12:27,710 [myid:1] - INFO [LearnerHandler-/127.0.0.1:50178:LearnerHandler@441] - Sending snapshot last zxid of peer is 0x0 zxid of leader is 0x200000001sent zxid of db as 0x200000001
2017-01-24 22:12:27,710 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11233:Learner@329] - Getting a snapshot from leader
2017-01-24 22:12:27,712 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11233:FileTxnSnapLog@281] - Snapshotting: 0x200000001 to /data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test2397521463065956250.junit.dir/data/version-2/snapshot.200000001
2017-01-24 22:12:27,713 [myid:1] - INFO [LearnerHandler-/127.0.0.1:50178:LearnerHandler@477] - Received NEWLEADER-ACK message from 2
2017-01-24 22:12:27,935 [myid:2] - INFO [WorkerSender[myid=2]:FastLeaderElection$Messenger$WorkerSender@370] - WorkerSender is down
2017-01-24 22:12:27,959 [myid:] - INFO [Thread-12:FourLetterWordMain@43] - connecting to 127.0.0.1 11233
2017-01-24 22:12:27,960 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45810
2017-01-24 22:12:27,960 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:45810
2017-01-24 22:12:27,961 [myid:2] - INFO [Thread-40:NIOServerCnxn$StatCommand@655] - Stat command output
2017-01-24 22:12:27,962 [myid:2] - INFO [Thread-40:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45810 (no session established for client)
2017-01-24 22:12:27,964 [myid:2] - WARN [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11233:Follower@119] - Got zxid 0x200000002 expected 0x1
2017-01-24 22:12:27,964 [myid:2] - INFO [SyncThread:2:FileTxnLog@199] - Creating new log file: log.200000002
2017-01-24 22:12:27,967 [myid:] - INFO [Thread-12:QuorumAuthUpgradeTest@229] - Restarting server myid=0
2017-01-24 22:12:27,967 [myid:] - INFO [Thread-12:QuorumBase@314] - Shutting down quorum peer QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11230
2017-01-24 22:12:27,967 [myid:] - INFO [Thread-12:Leader@491] - Shutting down
2017-01-24 22:12:27,967 [myid:] - INFO [Thread-12:Leader@497] - Shutdown called
java.lang.Exception: shutdown Leader! reason: quorum Peer shutdown
at org.apache.zookeeper.server.quorum.Leader.shutdown(Leader.java:497)
at org.apache.zookeeper.server.quorum.QuorumPeer.shutdown(QuorumPeer.java:893)
at org.apache.zookeeper.test.QuorumBase.shutdown(QuorumBase.java:315)
at org.apache.zookeeper.server.quorum.QuorumPeerTestBase$TestQPMain.shutdown(QuorumPeerTestBase.java:59)
at org.apache.zookeeper.server.quorum.QuorumPeerTestBase$MainThread.shutdown(QuorumPeerTestBase.java:152)
at org.apache.zookeeper.server.quorum.auth.QuorumAuthTestBase.shutdown(QuorumAuthTestBase.java:138)
at org.apache.zookeeper.server.quorum.auth.QuorumAuthUpgradeTest.restartServer(QuorumAuthUpgradeTest.java:230)
at org.apache.zookeeper.server.quorum.auth.QuorumAuthUpgradeTest.testRollingUpgrade(QuorumAuthUpgradeTest.java:203)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:52)
at org.junit.internal.runners.statements.FailOnTimeout$1.run(FailOnTimeout.java:28)
2017-01-24 22:12:27,968 [myid:] - INFO [Thread-12:ZooKeeperServer@419] - shutting down
2017-01-24 22:12:27,968 [myid:] - INFO [Thread-12:SessionTrackerImpl@225] - Shutting down
2017-01-24 22:12:27,968 [myid:] - INFO [Thread-12:PrepRequestProcessor@761] - Shutting down
2017-01-24 22:12:27,968 [myid:] - INFO [Thread-12:ProposalRequestProcessor@88] - Shutting down
2017-01-24 22:12:27,968 [myid:] - INFO [Thread-12:CommitProcessor@181] - Shutting down
2017-01-24 22:12:27,969 [myid:1] - INFO [CommitProcessor:1:CommitProcessor@150] - CommitProcessor exited loop!
2017-01-24 22:12:27,969 [myid:] - INFO [Thread-12:Leader$ToBeAppliedRequestProcessor@656] - Shutting down
2017-01-24 22:12:27,969 [myid:] - INFO [Thread-12:FinalRequestProcessor@415] - shutdown of request processor complete
2017-01-24 22:12:27,969 [myid:] - INFO [Thread-12:SyncRequestProcessor@175] - Shutting down
2017-01-24 22:12:27,968 [myid:1] - INFO [Thread-32:Leader$LearnerCnxAcceptor@318] - exception while shutting down acceptor: java.net.SocketException: Socket closed
2017-01-24 22:12:27,971 [myid:1] - INFO [ProcessThread(sid:1 cport:-1)::PrepRequestProcessor@143] - PrepRequestProcessor exited loop!
2017-01-24 22:12:27,971 [myid:1] - INFO [SyncThread:1:SyncRequestProcessor@155] - SyncRequestProcessor exited!
2017-01-24 22:12:27,972 [myid:1] - WARN [LearnerHandler-/127.0.0.1:50170:LearnerHandler@598] - ******* GOODBYE /127.0.0.1:50170 ********
2017-01-24 22:12:27,972 [myid:1] - WARN [LearnerHandler-/127.0.0.1:50170:LearnerHandler@610] - Ignoring unexpected exception
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1219)
at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:340)
at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:338)
at org.apache.zookeeper.server.quorum.LearnerHandler.shutdown(LearnerHandler.java:608)
at org.apache.zookeeper.server.quorum.LearnerHandler.run(LearnerHandler.java:601)
2017-01-24 22:12:27,973 [myid:1] - WARN [LearnerHandler-/127.0.0.1:50178:LearnerHandler@598] - ******* GOODBYE /127.0.0.1:50178 ********
2017-01-24 22:12:27,973 [myid:1] - WARN [LearnerHandler-/127.0.0.1:50178:LearnerHandler@610] - Ignoring unexpected exception
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1219)
at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:340)
at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:338)
at org.apache.zookeeper.server.quorum.LearnerHandler.shutdown(LearnerHandler.java:608)
at org.apache.zookeeper.server.quorum.LearnerHandler.run(LearnerHandler.java:601)
2017-01-24 22:12:27,973 [myid:1] - WARN [LearnerHandler-/127.0.0.1:50169:LearnerHandler@598] - ******* GOODBYE /127.0.0.1:50169 ********
2017-01-24 22:12:27,974 [myid:0] - WARN [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11227:Follower@90] - Exception when following the leader
java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:392)
at org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63)
at org.apache.zookeeper.server.quorum.QuorumPacket.deserialize(QuorumPacket.java:83)
at org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:99)
at org.apache.zookeeper.server.quorum.Learner.readPacket(Learner.java:152)
at org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:86)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:851)
2017-01-24 22:12:27,974 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11227:Follower@167] - shutdown called
java.lang.Exception: shutdown Follower
at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:167)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:855)
2017-01-24 22:12:27,974 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11230:NIOServerCnxnFactory@224] - NIOServerCnxn factory exited run method
2017-01-24 22:12:27,975 [myid:2] - WARN [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11233:Follower@90] - Exception when following the leader
java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:392)
at org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63)
at org.apache.zookeeper.server.quorum.QuorumPacket.deserialize(QuorumPacket.java:83)
at org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:99)
at org.apache.zookeeper.server.quorum.Learner.readPacket(Learner.java:152)
at org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:86)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:851)
2017-01-24 22:12:27,975 [myid:1] - WARN [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11230:QuorumPeer@867] - Unexpected exception
java.lang.InterruptedException: sleep interrupted
at java.lang.Thread.sleep(Native Method)
at org.apache.zookeeper.server.quorum.Leader.lead(Leader.java:451)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:864)
2017-01-24 22:12:27,975 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11230:Leader@491] - Shutting down
2017-01-24 22:12:27,975 [myid:1] - WARN [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11230:QuorumPeer@879] - QuorumPeer main thread exited
2017-01-24 22:12:27,975 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11233:Follower@167] - shutdown called
java.lang.Exception: shutdown Follower
at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:167)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:855)
2017-01-24 22:12:27,975 [myid:1] - ERROR [localhost/127.0.0.1:11232:QuorumCnxManager$Listener@715] - Exception while listening
java.net.SocketException: Socket closed
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
at java.net.ServerSocket.implAccept(ServerSocket.java:530)
at java.net.ServerSocket.accept(ServerSocket.java:498)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$Listener.run(QuorumCnxManager.java:696)
2017-01-24 22:12:27,976 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11233:FollowerZooKeeperServer@139] - Shutting down
2017-01-24 22:12:27,977 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11233:ZooKeeperServer@419] - shutting down
2017-01-24 22:12:27,977 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11233:FollowerRequestProcessor@105] - Shutting down
2017-01-24 22:12:27,978 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11233:CommitProcessor@181] - Shutting down
2017-01-24 22:12:27,978 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11233:FinalRequestProcessor@415] - shutdown of request processor complete
2017-01-24 22:12:27,977 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11227:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:42834 which had sessionid 0x159d440f0ed0000
2017-01-24 22:12:27,977 [myid:1] - WARN [RecvWorker:0:QuorumCnxManager$RecvWorker@980] - Connection broken for id 0, my id = 1, error =
java.net.SocketException: Socket closed
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at java.io.BufferedInputStream.read(BufferedInputStream.java:254)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:965)
2017-01-24 22:12:27,979 [myid:1] - WARN [RecvWorker:0:QuorumCnxManager$RecvWorker@983] - Interrupting SendWorker
2017-01-24 22:12:27,977 [myid:0] - WARN [RecvWorker:1:QuorumCnxManager$RecvWorker@980] - Connection broken for id 1, my id = 0, error =
java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:392)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:965)
2017-01-24 22:12:27,977 [myid:1] - WARN [SendWorker:0:QuorumCnxManager$SendWorker@896] - Interrupted while waiting for message on queue
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1049)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:73)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:884)
2017-01-24 22:12:27,980 [myid:1] - WARN [SendWorker:0:QuorumCnxManager$SendWorker@906] - Send worker leaving thread
2017-01-24 22:12:27,979 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11227:FollowerZooKeeperServer@139] - Shutting down
2017-01-24 22:12:27,980 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11227:ZooKeeperServer@419] - shutting down
2017-01-24 22:12:27,980 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11227:FollowerRequestProcessor@105] - Shutting down
2017-01-24 22:12:27,980 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11227:CommitProcessor@181] - Shutting down
2017-01-24 22:12:27,980 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11227:FinalRequestProcessor@415] - shutdown of request processor complete
2017-01-24 22:12:27,980 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11227:SyncRequestProcessor@175] - Shutting down
2017-01-24 22:12:27,979 [myid:] - INFO [Thread-12-SendThread(localhost:11227):ClientCnxn$SendThread@1098] - Unable to read additional data from server sessionid 0x159d440f0ed0000, likely server has closed socket, closing socket connection and attempting reconnect
2017-01-24 22:12:27,979 [myid:1] - WARN [RecvWorker:2:QuorumCnxManager$RecvWorker@980] - Connection broken for id 2, my id = 1, error =
java.net.SocketException: Socket closed
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at java.io.BufferedInputStream.read(BufferedInputStream.java:254)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:965)
2017-01-24 22:12:27,981 [myid:1] - WARN [RecvWorker:2:QuorumCnxManager$RecvWorker@983] - Interrupting SendWorker
2017-01-24 22:12:27,978 [myid:] - INFO [Thread-12:QuorumBase@318] - Shutting down leader election QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11230
2017-01-24 22:12:27,978 [myid:1] - WARN [SendWorker:2:QuorumCnxManager$SendWorker@896] - Interrupted while waiting for message on queue
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1049)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:73)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:884)
2017-01-24 22:12:27,981 [myid:1] - WARN [SendWorker:2:QuorumCnxManager$SendWorker@906] - Send worker leaving thread
2017-01-24 22:12:27,978 [myid:2] - WARN [RecvWorker:1:QuorumCnxManager$RecvWorker@980] - Connection broken for id 1, my id = 2, error =
java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:392)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:965)
2017-01-24 22:12:27,982 [myid:2] - WARN [RecvWorker:1:QuorumCnxManager$RecvWorker@983] - Interrupting SendWorker
2017-01-24 22:12:27,978 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11233:SyncRequestProcessor@175] - Shutting down
2017-01-24 22:12:27,978 [myid:2] - INFO [CommitProcessor:2:CommitProcessor@150] - CommitProcessor exited loop!
2017-01-24 22:12:27,978 [myid:2] - INFO [FollowerRequestProcessor:2:FollowerRequestProcessor@95] - FollowerRequestProcessor exited loop!
2017-01-24 22:12:27,982 [myid:2] - INFO [SyncThread:2:SyncRequestProcessor@155] - SyncRequestProcessor exited!
2017-01-24 22:12:27,982 [myid:2] - WARN [SendWorker:1:QuorumCnxManager$SendWorker@896] - Interrupted while waiting for message on queue
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1049)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:73)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:884)
2017-01-24 22:12:27,986 [myid:2] - WARN [SendWorker:1:QuorumCnxManager$SendWorker@906] - Send worker leaving thread
2017-01-24 22:12:27,981 [myid:] - INFO [Thread-12:QuorumBase@323] - Waiting for QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11230 to exit thread
2017-01-24 22:12:27,980 [myid:0] - INFO [SyncThread:0:SyncRequestProcessor@155] - SyncRequestProcessor exited!
2017-01-24 22:12:27,980 [myid:0] - INFO [CommitProcessor:0:CommitProcessor@150] - CommitProcessor exited loop!
2017-01-24 22:12:27,980 [myid:0] - INFO [FollowerRequestProcessor:0:FollowerRequestProcessor@95] - FollowerRequestProcessor exited loop!
2017-01-24 22:12:27,979 [myid:0] - WARN [RecvWorker:1:QuorumCnxManager$RecvWorker@983] - Interrupting SendWorker
2017-01-24 22:12:27,987 [myid:0] - WARN [SendWorker:1:QuorumCnxManager$SendWorker@896] - Interrupted while waiting for message on queue
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1049)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:73)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:884)
2017-01-24 22:12:27,986 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11227:QuorumPeer@781] - LOOKING
2017-01-24 22:12:27,985 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11233:QuorumPeer@781] - LOOKING
2017-01-24 22:12:27,988 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11233:FileSnap@83] - Reading snapshot /data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test2397521463065956250.junit.dir/data/version-2/snapshot.200000001
2017-01-24 22:12:27,988 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11227:FileSnap@83] - Reading snapshot /data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test6261823493417515862.junit.dir/data/version-2/snapshot.100000003
2017-01-24 22:12:27,990 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11233:FastLeaderElection@744] - New election. My id = 2, proposed zxid=0x200000002
2017-01-24 22:12:27,990 [myid:0] - WARN [SendWorker:1:QuorumCnxManager$SendWorker@906] - Send worker leaving thread
2017-01-24 22:12:27,991 [myid:] - INFO [Thread-12:QuorumPeerTestBase$MainThread@81] - id = 1 tmpDir = /data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test4772652757083079773.junit.dir clientPort = 11230
2017-01-24 22:12:27,991 [myid:0] - INFO [WorkerReceiver[myid=0]:FastLeaderElection@542] - Notification: 2 (n.leader), 0x200000002 (n.zxid), 0x3 (n.round), LOOKING (n.state), 2 (n.sid), 0x2 (n.peerEPoch), LOOKING (my state)
2017-01-24 22:12:27,992 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11227:FastLeaderElection@744] - New election. My id = 0, proposed zxid=0x200000002
2017-01-24 22:12:27,992 [myid:2] - WARN [WorkerSender[myid=2]:QuorumCnxManager@559] - Cannot open channel to 1 at election address localhost/127.0.0.1:11232
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:538)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:514)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:393)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:365)
at java.lang.Thread.run(Thread.java:745)
2017-01-24 22:12:27,992 [myid:] - INFO [Thread-41:QuorumPeerConfig@111] - Reading configuration from: /data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test4772652757083079773.junit.dir/zoo.cfg
2017-01-24 22:12:27,992 [myid:0] - WARN [WorkerSender[myid=0]:QuorumCnxManager@559] - Cannot open channel to 1 at election address localhost/127.0.0.1:11232
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:538)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:514)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:393)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:365)
at java.lang.Thread.run(Thread.java:745)
2017-01-24 22:12:27,992 [myid:0] - INFO [WorkerReceiver[myid=0]:FastLeaderElection@542] - Notification: 0 (n.leader), 0x200000002 (n.zxid), 0x3 (n.round), LOOKING (n.state), 0 (n.sid), 0x2 (n.peerEPoch), LOOKING (my state)
2017-01-24 22:12:27,994 [myid:0] - INFO [WorkerReceiver[myid=0]:FastLeaderElection@542] - Notification: 2 (n.leader), 0x200000002 (n.zxid), 0x3 (n.round), LOOKING (n.state), 0 (n.sid), 0x2 (n.peerEPoch), LOOKING (my state)
2017-01-24 22:12:27,994 [myid:0] - WARN [WorkerSender[myid=0]:QuorumCnxManager@559] - Cannot open channel to 1 at election address localhost/127.0.0.1:11232
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:538)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:514)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:393)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:365)
at java.lang.Thread.run(Thread.java:745)
2017-01-24 22:12:27,993 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@542] - Notification: 2 (n.leader), 0x200000002 (n.zxid), 0x3 (n.round), LOOKING (n.state), 2 (n.sid), 0x2 (n.peerEPoch), LOOKING (my state)
2017-01-24 22:12:27,994 [myid:] - INFO [Thread-41:QuorumPeerConfig@374] - Defaulting to majority quorums
2017-01-24 22:12:27,995 [myid:] - INFO [Thread-12:FourLetterWordMain@43] - connecting to 127.0.0.1 11230
2017-01-24 22:12:27,995 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@542] - Notification: 2 (n.leader), 0x200000002 (n.zxid), 0x3 (n.round), LOOKING (n.state), 0 (n.sid), 0x2 (n.peerEPoch), LOOKING (my state)
2017-01-24 22:12:27,995 [myid:1] - INFO [Thread-41:DatadirCleanupManager@78] - autopurge.snapRetainCount set to 3
2017-01-24 22:12:27,995 [myid:1] - INFO [Thread-41:DatadirCleanupManager@79] - autopurge.purgeInterval set to 0
2017-01-24 22:12:27,995 [myid:1] - INFO [Thread-41:DatadirCleanupManager@101] - Purge task is not scheduled.
2017-01-24 22:12:27,996 [myid:1] - WARN [Thread-41:QuorumPeerMain@129] - Unable to register log4j JMX control
javax.management.InstanceAlreadyExistsException: log4j:hiearchy=default
at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:437)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1898)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:966)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324)
at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
at org.apache.zookeeper.jmx.ManagedUtil.registerLog4jMBeans(ManagedUtil.java:53)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:127)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:116)
at org.apache.zookeeper.server.quorum.QuorumPeerTestBase$MainThread.run(QuorumPeerTestBase.java:140)
at java.lang.Thread.run(Thread.java:745)
2017-01-24 22:12:28,001 [myid:1] - INFO [Thread-41:QuorumPeerMain@132] - Starting quorum peer
2017-01-24 22:12:28,001 [myid:1] - INFO [Thread-41:NIOServerCnxnFactory@94] - binding to port 0.0.0.0/0.0.0.0:11230
2017-01-24 22:12:28,002 [myid:1] - INFO [Thread-41:QuorumPeer@1048] - minSessionTimeout set to -1
2017-01-24 22:12:28,002 [myid:1] - INFO [Thread-41:QuorumPeer@1059] - maxSessionTimeout set to -1
2017-01-24 22:12:28,002 [myid:1] - INFO [Thread-41:QuorumPeer@1279] - quorum.auth.enableSasl set to true
2017-01-24 22:12:28,002 [myid:1] - INFO [Thread-41:QuorumPeer@1264] - quorum.auth.serverRequireSasl set to false
2017-01-24 22:12:28,002 [myid:1] - INFO [Thread-41:QuorumPeer@1270] - quorum.auth.learnerRequireSasl set to true
2017-01-24 22:12:28,002 [myid:1] - INFO [Thread-41:QuorumPeer@1286] - quorum.auth.kerberos.servicePrincipal set to zkquorum/localhost
2017-01-24 22:12:28,002 [myid:1] - INFO [Thread-41:QuorumPeer@1298] - quorum.auth.server.saslLoginContext set to QuorumServer
2017-01-24 22:12:28,003 [myid:1] - INFO [Thread-41:QuorumPeer@1292] - quorum.auth.learner.saslLoginContext set to QuorumLearner
2017-01-24 22:12:28,003 [myid:1] - INFO [Thread-41:QuorumPeer@1306] - quorum.cnxn.threads.size set to 20
2017-01-24 22:12:28,003 [myid:1] - INFO [Thread-41:Login@294] - QuorumServer successfully logged in.
2017-01-24 22:12:28,003 [myid:1] - INFO [Thread-41:Login@294] - QuorumLearner successfully logged in.
2017-01-24 22:12:28,004 [myid:1] - INFO [Thread-41:QuorumPeer@540] - currentEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
2017-01-24 22:12:27,995 [myid:] - INFO [Thread-12:ClientBase@246] - server 127.0.0.1:11230 not up java.net.ConnectException: Connection refused
2017-01-24 22:12:28,000 [myid:2] - INFO [SessionTracker:SessionTrackerImpl@162] - SessionTrackerImpl exited loop!
2017-01-24 22:12:28,000 [myid:1] - INFO [SessionTracker:SessionTrackerImpl@162] - SessionTrackerImpl exited loop!
2017-01-24 22:12:28,006 [myid:1] - INFO [Thread-41:QuorumPeer@555] - acceptedEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
2017-01-24 22:12:28,012 [myid:1] - INFO [Thread-42:QuorumCnxManager$Listener@691] - My election bind port: 0.0.0.0/0.0.0.0:11232
2017-01-24 22:12:28,013 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11230:QuorumPeer@781] - LOOKING
2017-01-24 22:12:28,013 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11230:FastLeaderElection@744] - New election. My id = 1, proposed zxid=0x0
2017-01-24 22:12:28,014 [myid:0] - INFO [localhost/127.0.0.1:11229:QuorumCnxManager$Listener@698] - Received connection request /127.0.0.1:51859
2017-01-24 22:12:28,014 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@542] - Notification: 1 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)
2017-01-24 22:12:28,014 [myid:2] - INFO [localhost/127.0.0.1:11235:QuorumCnxManager$Listener@698] - Received connection request /127.0.0.1:50365
2017-01-24 22:12:28,021 [myid:1] - INFO [QuorumConnectionThread-[myid=1]-2:SecurityUtils@68] - QuorumLearner will use DIGEST-MD5 as SASL mechanism.
2017-01-24 22:12:28,021 [myid:1] - INFO [QuorumConnectionThread-[myid=1]-1:SecurityUtils@68] - QuorumLearner will use DIGEST-MD5 as SASL mechanism.
2017-01-24 22:12:28,148 [myid:0] - INFO [QuorumConnectionThread-[myid=0]-1:SaslQuorumServerCallbackHandler@143] - Successfully authenticated learner: authenticationID=test; authorizationID=test.
2017-01-24 22:12:28,148 [myid:2] - INFO [QuorumConnectionThread-[myid=2]-3:SaslQuorumServerCallbackHandler@143] - Successfully authenticated learner: authenticationID=test; authorizationID=test.
2017-01-24 22:12:28,149 [myid:0] - INFO [QuorumConnectionThread-[myid=0]-1:SaslQuorumAuthServer@114] - Successfully completed the authentication using SASL. learner addr: /127.0.0.1:51859
2017-01-24 22:12:28,149 [myid:2] - INFO [QuorumConnectionThread-[myid=2]-3:SaslQuorumAuthServer@114] - Successfully completed the authentication using SASL. learner addr: /127.0.0.1:50365
2017-01-24 22:12:28,150 [myid:1] - INFO [QuorumConnectionThread-[myid=1]-2:SaslQuorumAuthLearner@151] - Successfully completed the authentication using SASL. server addr: localhost/127.0.0.1:11235, status: SUCCESS
2017-01-24 22:12:28,150 [myid:1] - INFO [QuorumConnectionThread-[myid=1]-2:QuorumCnxManager@331] - Have smaller server identifier, so dropping the connection: (2, 1)
2017-01-24 22:12:28,152 [myid:1] - INFO [QuorumConnectionThread-[myid=1]-1:SaslQuorumAuthLearner@151] - Successfully completed the authentication using SASL. server addr: localhost/127.0.0.1:11229, status: SUCCESS
2017-01-24 22:12:28,157 [myid:1] - INFO [localhost/127.0.0.1:11232:QuorumCnxManager$Listener@698] - Received connection request /127.0.0.1:45208
2017-01-24 22:12:28,157 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@542] - Notification: 2 (n.leader), 0x200000002 (n.zxid), 0x3 (n.round), LOOKING (n.state), 0 (n.sid), 0x2 (n.peerEPoch), LOOKING (my state)
2017-01-24 22:12:28,158 [myid:0] - INFO [WorkerReceiver[myid=0]:FastLeaderElection@542] - Notification: 1 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)
2017-01-24 22:12:28,158 [myid:2] - INFO [QuorumConnectionThread-[myid=2]-2:SaslQuorumAuthLearner@79] - Skipping SASL authentication as quorum.auth.learnerRequireSasl=false
2017-01-24 22:12:28,159 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@542] - Notification: 2 (n.leader), 0x200000002 (n.zxid), 0x3 (n.round), LOOKING (n.state), 1 (n.sid), 0x2 (n.peerEPoch), LOOKING (my state)
2017-01-24 22:12:28,160 [myid:2] - INFO [localhost/127.0.0.1:11235:QuorumCnxManager$Listener@698] - Received connection request /127.0.0.1:50367
2017-01-24 22:12:28,160 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@542] - Notification: 2 (n.leader), 0x200000002 (n.zxid), 0x3 (n.round), LOOKING (n.state), 0 (n.sid), 0x2 (n.peerEPoch), LOOKING (my state)
2017-01-24 22:12:28,160 [myid:0] - INFO [WorkerReceiver[myid=0]:FastLeaderElection@542] - Notification: 2 (n.leader), 0x200000002 (n.zxid), 0x3 (n.round), LOOKING (n.state), 1 (n.sid), 0x2 (n.peerEPoch), LOOKING (my state)
2017-01-24 22:12:28,160 [myid:1] - INFO [QuorumConnectionThread-[myid=1]-1:SecurityUtils@68] - QuorumLearner will use DIGEST-MD5 as SASL mechanism.
2017-01-24 22:12:28,167 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@542] - Notification: 2 (n.leader), 0x200000002 (n.zxid), 0x3 (n.round), LOOKING (n.state), 1 (n.sid), 0x2 (n.peerEPoch), LOOKING (my state)
2017-01-24 22:12:28,167 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@542] - Notification: 2 (n.leader), 0x200000002 (n.zxid), 0x3 (n.round), LOOKING (n.state), 2 (n.sid), 0x2 (n.peerEPoch), LOOKING (my state)
2017-01-24 22:12:28,179 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@340] - WorkerReceiver is down
2017-01-24 22:12:28,180 [myid:2] - INFO [QuorumConnectionThread-[myid=2]-2:SaslQuorumServerCallbackHandler@143] - Successfully authenticated learner: authenticationID=test; authorizationID=test.
2017-01-24 22:12:28,180 [myid:2] - INFO [QuorumConnectionThread-[myid=2]-2:SaslQuorumAuthServer@114] - Successfully completed the authentication using SASL. learner addr: /127.0.0.1:50367
2017-01-24 22:12:28,181 [myid:1] - WARN [RecvWorker:2:QuorumCnxManager$RecvWorker@980] - Connection broken for id 2, my id = 1, error =
java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:392)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:965)
2017-01-24 22:12:28,181 [myid:1] - WARN [RecvWorker:2:QuorumCnxManager$RecvWorker@983] - Interrupting SendWorker
2017-01-24 22:12:28,181 [myid:2] - WARN [RecvWorker:1:QuorumCnxManager$RecvWorker@980] - Connection broken for id 1, my id = 2, error =
java.net.SocketException: Socket closed
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at java.io.BufferedInputStream.read(BufferedInputStream.java:254)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:965)
2017-01-24 22:12:28,182 [myid:2] - WARN [RecvWorker:1:QuorumCnxManager$RecvWorker@983] - Interrupting SendWorker
2017-01-24 22:12:28,183 [myid:2] - INFO [QuorumConnectionThread-[myid=2]-3:SaslQuorumAuthLearner@79] - Skipping SASL authentication as quorum.auth.learnerRequireSasl=false
2017-01-24 22:12:28,181 [myid:2] - WARN [SendWorker:1:QuorumCnxManager$SendWorker@896] - Interrupted while waiting for message on queue
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1049)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:73)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:884)
2017-01-24 22:12:28,184 [myid:2] - WARN [SendWorker:1:QuorumCnxManager$SendWorker@906] - Send worker leaving thread
2017-01-24 22:12:28,182 [myid:1] - WARN [SendWorker:2:QuorumCnxManager$SendWorker@896] - Interrupted while waiting for message on queue
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1049)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:73)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:884)
2017-01-24 22:12:28,184 [myid:1] - WARN [SendWorker:2:QuorumCnxManager$SendWorker@906] - Send worker leaving thread
2017-01-24 22:12:28,182 [myid:1] - INFO [QuorumConnectionThread-[myid=1]-1:SaslQuorumAuthLearner@151] - Successfully completed the authentication using SASL. server addr: localhost/127.0.0.1:11235, status: SUCCESS
2017-01-24 22:12:28,182 [myid:1] - INFO [localhost/127.0.0.1:11232:QuorumCnxManager$Listener@698] - Received connection request /127.0.0.1:45210
2017-01-24 22:12:28,184 [myid:1] - INFO [QuorumConnectionThread-[myid=1]-1:QuorumCnxManager@331] - Have smaller server identifier, so dropping the connection: (2, 1)
2017-01-24 22:12:28,189 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@542] - Notification: 2 (n.leader), 0x200000002 (n.zxid), 0x3 (n.round), LOOKING (n.state), 1 (n.sid), 0x2 (n.peerEPoch), LOOKING (my state)
2017-01-24 22:12:28,189 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@542] - Notification: 2 (n.leader), 0x200000002 (n.zxid), 0x3 (n.round), LOOKING (n.state), 2 (n.sid), 0x2 (n.peerEPoch), LOOKING (my state)
2017-01-24 22:12:28,193 [myid:1] - ERROR [LearnerHandler-/127.0.0.1:57759:LearnerHandler@585] - Unexpected exception causing shutdown while sock still open
java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:196)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at java.io.BufferedInputStream.read(BufferedInputStream.java:254)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63)
at org.apache.zookeeper.server.quorum.QuorumPacket.deserialize(QuorumPacket.java:83)
at org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:99)
at org.apache.zookeeper.server.quorum.LearnerHandler.run(LearnerHandler.java:499)
2017-01-24 22:12:28,193 [myid:1] - WARN [LearnerHandler-/127.0.0.1:57759:LearnerHandler@598] - ******* GOODBYE /127.0.0.1:57759 ********
2017-01-24 22:12:28,212 [myid:2] - INFO [WorkerSender[myid=2]:FastLeaderElection$Messenger$WorkerSender@370] - WorkerSender is down
2017-01-24 22:12:28,215 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@340] - WorkerReceiver is down
2017-01-24 22:12:28,256 [myid:] - INFO [Thread-12:FourLetterWordMain@43] - connecting to 127.0.0.1 11230
2017-01-24 22:12:28,257 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11230:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:52351
2017-01-24 22:12:28,257 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11230:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:52351
2017-01-24 22:12:28,258 [myid:1] - INFO [Thread-43:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:52351 (no session established for client)
2017-01-24 22:12:28,360 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11227:QuorumPeer@849] - FOLLOWING
2017-01-24 22:12:28,361 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11227:ZooKeeperServer@162] - Created server with tickTime 4000 minSessionTimeout 8000 maxSessionTimeout 80000 datadir /data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test6261823493417515862.junit.dir/data/version-2 snapdir /data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test6261823493417515862.junit.dir/data/version-2
2017-01-24 22:12:28,361 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11227:Follower@64] - FOLLOWING - LEADER ELECTION TOOK - 373
2017-01-24 22:12:28,363 [myid:0] - WARN [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11227:Learner@233] - Unexpected exception, tries=0, connecting to localhost/127.0.0.1:11234
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.server.quorum.Learner.connectToLeader(Learner.java:225)
at org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:72)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:851)
2017-01-24 22:12:28,390 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11233:QuorumPeer@861] - LEADING
2017-01-24 22:12:28,390 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11233:ZooKeeperServer@162] - Created server with tickTime 4000 minSessionTimeout 8000 maxSessionTimeout 80000 datadir /data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test2397521463065956250.junit.dir/data/version-2 snapdir /data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test2397521463065956250.junit.dir/data/version-2
2017-01-24 22:12:28,390 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11233:Leader@356] - LEADING - LEADER ELECTION TOOK - 402
2017-01-24 22:12:28,390 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11230:QuorumPeer@849] - FOLLOWING
2017-01-24 22:12:28,390 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11230:ZooKeeperServer@162] - Created server with tickTime 4000 minSessionTimeout 8000 maxSessionTimeout 80000 datadir /data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test4772652757083079773.junit.dir/data/version-2 snapdir /data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test4772652757083079773.junit.dir/data/version-2
2017-01-24 22:12:28,390 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11230:Follower@64] - FOLLOWING - LEADER ELECTION TOOK - 377
2017-01-24 22:12:28,392 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11230:SecurityUtils@68] - QuorumLearner will use DIGEST-MD5 as SASL mechanism.
2017-01-24 22:12:28,392 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11233:FileSnap@83] - Reading snapshot /data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test2397521463065956250.junit.dir/data/version-2/snapshot.200000001
2017-01-24 22:12:28,393 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11233:FileTxnSnapLog@281] - Snapshotting: 0x200000002 to /data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test2397521463065956250.junit.dir/data/version-2/snapshot.200000002
2017-01-24 22:12:28,396 [myid:2] - INFO [Thread-44:SaslQuorumServerCallbackHandler@143] - Successfully authenticated learner: authenticationID=test; authorizationID=test.
2017-01-24 22:12:28,396 [myid:2] - INFO [Thread-44:SaslQuorumAuthServer@114] - Successfully completed the authentication using SASL. learner addr: /127.0.0.1:42592
2017-01-24 22:12:28,398 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11230:SaslQuorumAuthLearner@151] - Successfully completed the authentication using SASL. server addr: localhost/127.0.0.1:11234, status: SUCCESS
2017-01-24 22:12:28,398 [myid:2] - INFO [LearnerHandler-/127.0.0.1:42592:LearnerHandler@287] - Follower sid: 1 : info : org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@b6804a5
2017-01-24 22:12:28,404 [myid:2] - INFO [LearnerHandler-/127.0.0.1:42592:LearnerHandler@342] - Synchronizing with Follower sid: 1 maxCommittedLog=0x0 minCommittedLog=0x0 peerLastZxid=0x0
2017-01-24 22:12:28,404 [myid:2] - INFO [LearnerHandler-/127.0.0.1:42592:LearnerHandler@441] - Sending snapshot last zxid of peer is 0x0 zxid of leader is 0x300000000sent zxid of db as 0x200000002
2017-01-24 22:12:28,405 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11230:Learner@329] - Getting a snapshot from leader
2017-01-24 22:12:28,411 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11230:FileTxnSnapLog@281] - Snapshotting: 0x200000002 to /data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test4772652757083079773.junit.dir/data/version-2/snapshot.200000002
2017-01-24 22:12:28,413 [myid:2] - INFO [LearnerHandler-/127.0.0.1:42592:LearnerHandler@477] - Received NEWLEADER-ACK message from 1
2017-01-24 22:12:28,413 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11233:Leader@934] - Have quorum of supporters, sids: [ 1,2 ]; starting up and setting last processed zxid: 0x300000000
2017-01-24 22:12:28,508 [myid:] - INFO [Thread-12:FourLetterWordMain@43] - connecting to 127.0.0.1 11230
2017-01-24 22:12:28,509 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11230:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:52354
2017-01-24 22:12:28,509 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11230:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:52354
2017-01-24 22:12:28,509 [myid:1] - INFO [Thread-46:NIOServerCnxn$StatCommand@655] - Stat command output
2017-01-24 22:12:28,510 [myid:1] - INFO [Thread-46:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:52354 (no session established for client)
2017-01-24 22:12:28,648 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11224:Leader@491] - Shutting down
2017-01-24 22:12:28,648 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11224:Leader@497] - Shutdown called
java.lang.Exception: shutdown Leader! reason: Not sufficient followers synced, only synced with sids: [ 1 ]
at org.apache.zookeeper.server.quorum.Leader.shutdown(Leader.java:497)
at org.apache.zookeeper.server.quorum.Leader.lead(Leader.java:472)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:864)
2017-01-24 22:12:28,648 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11224:ZooKeeperServer@419] - shutting down
2017-01-24 22:12:28,648 [myid:1] - INFO [Thread-8:Leader$LearnerCnxAcceptor@318] - exception while shutting down acceptor: java.net.SocketException: Socket closed
2017-01-24 22:12:28,648 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11224:SessionTrackerImpl@225] - Shutting down
2017-01-24 22:12:28,649 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11224:PrepRequestProcessor@761] - Shutting down
2017-01-24 22:12:28,649 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11224:ProposalRequestProcessor@88] - Shutting down
2017-01-24 22:12:28,649 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11224:CommitProcessor@181] - Shutting down
2017-01-24 22:12:28,649 [myid:1] - INFO [ProcessThread(sid:1 cport:-1)::PrepRequestProcessor@143] - PrepRequestProcessor exited loop!
2017-01-24 22:12:28,649 [myid:1] - INFO [CommitProcessor:1:CommitProcessor@150] - CommitProcessor exited loop!
2017-01-24 22:12:28,649 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11224:Leader$ToBeAppliedRequestProcessor@656] - Shutting down
2017-01-24 22:12:28,649 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11224:FinalRequestProcessor@415] - shutdown of request processor complete
2017-01-24 22:12:28,649 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11224:SyncRequestProcessor@175] - Shutting down
2017-01-24 22:12:28,649 [myid:1] - INFO [SyncThread:1:SyncRequestProcessor@155] - SyncRequestProcessor exited!
2017-01-24 22:12:28,650 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11224:QuorumPeer@781] - LOOKING
2017-01-24 22:12:28,650 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11224:FileSnap@83] - Reading snapshot /data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test4158500655544466289.junit.dir/data/version-2/snapshot.0
2017-01-24 22:12:28,651 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11224:FastLeaderElection@744] - New election. My id = 1, proposed zxid=0x100000003
2017-01-24 22:12:28,652 [myid:1] - WARN [WorkerSender[myid=1]:QuorumCnxManager@559] - Cannot open channel to 0 at election address localhost/127.0.0.1:11223
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:538)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:514)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:393)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:365)
at java.lang.Thread.run(Thread.java:745)
2017-01-24 22:12:28,652 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@542] - Notification: 1 (n.leader), 0x100000003 (n.zxid), 0x2 (n.round), LOOKING (n.state), 1 (n.sid), 0x1 (n.peerEPoch), LOOKING (my state)
2017-01-24 22:12:28,768 [myid:] - WARN [Thread-12-SendThread(localhost:11230):ClientCnxn$SendThread@957] - SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: '/data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test2999887027410032136.junit.dir/jaas.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.
2017-01-24 22:12:28,768 [myid:] - INFO [Thread-12-SendThread(localhost:11230):ClientCnxn$SendThread@975] - Opening socket connection to server localhost/127.0.0.1:11230
2017-01-24 22:12:28,769 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11230:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:52356
2017-01-24 22:12:28,769 [myid:] - INFO [Thread-12-SendThread(localhost:11230):ClientCnxn$SendThread@852] - Socket connection established, initiating session, client: /127.0.0.1:52356, server: localhost/127.0.0.1:11230
2017-01-24 22:12:28,769 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11230:ZooKeeperServer@832] - Client attempting to renew session 0x159d440f0ed0000 at /127.0.0.1:52356
2017-01-24 22:12:28,769 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11230:Learner@107] - Revalidating client: 0x159d440f0ed0000
2017-01-24 22:12:28,772 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11230:ZooKeeperServer@595] - Established session 0x159d440f0ed0000 with negotiated timeout 30000 for client /127.0.0.1:52356
2017-01-24 22:12:28,772 [myid:] - INFO [Thread-12-SendThread(localhost:11230):ClientCnxn$SendThread@1235] - Session establishment complete on server localhost/127.0.0.1:11230, sessionid = 0x159d440f0ed0000, negotiated timeout = 30000
2017-01-24 22:12:28,774 [myid:1] - WARN [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11230:Follower@119] - Got zxid 0x300000001 expected 0x1
2017-01-24 22:12:28,775 [myid:1] - INFO [SyncThread:1:FileTxnLog@199] - Creating new log file: log.300000001
2017-01-24 22:12:28,777 [myid:] - INFO [Thread-12:QuorumAuthUpgradeTest@229] - Restarting server myid=1
2017-01-24 22:12:28,777 [myid:] - INFO [Thread-12:QuorumBase@314] - Shutting down quorum peer QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11233
2017-01-24 22:12:28,777 [myid:] - INFO [Thread-12:Leader@491] - Shutting down
2017-01-24 22:12:28,777 [myid:] - INFO [Thread-12:Leader@497] - Shutdown called
java.lang.Exception: shutdown Leader! reason: quorum Peer shutdown
at org.apache.zookeeper.server.quorum.Leader.shutdown(Leader.java:497)
at org.apache.zookeeper.server.quorum.QuorumPeer.shutdown(QuorumPeer.java:893)
at org.apache.zookeeper.test.QuorumBase.shutdown(QuorumBase.java:315)
at org.apache.zookeeper.server.quorum.QuorumPeerTestBase$TestQPMain.shutdown(QuorumPeerTestBase.java:59)
at org.apache.zookeeper.server.quorum.QuorumPeerTestBase$MainThread.shutdown(QuorumPeerTestBase.java:152)
at org.apache.zookeeper.server.quorum.auth.QuorumAuthTestBase.shutdown(QuorumAuthTestBase.java:138)
at org.apache.zookeeper.server.quorum.auth.QuorumAuthUpgradeTest.restartServer(QuorumAuthUpgradeTest.java:230)
at org.apache.zookeeper.server.quorum.auth.QuorumAuthUpgradeTest.testRollingUpgrade(QuorumAuthUpgradeTest.java:204)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:52)
at org.junit.internal.runners.statements.FailOnTimeout$1.run(FailOnTimeout.java:28)
2017-01-24 22:12:28,778 [myid:] - INFO [Thread-12:ZooKeeperServer@419] - shutting down
2017-01-24 22:12:28,778 [myid:2] - INFO [Thread-44:Leader$LearnerCnxAcceptor@318] - exception while shutting down acceptor: java.net.SocketException: Socket closed
2017-01-24 22:12:28,778 [myid:] - INFO [Thread-12:SessionTrackerImpl@225] - Shutting down
2017-01-24 22:12:28,778 [myid:] - INFO [Thread-12:PrepRequestProcessor@761] - Shutting down
2017-01-24 22:12:28,779 [myid:] - INFO [Thread-12:ProposalRequestProcessor@88] - Shutting down
2017-01-24 22:12:28,779 [myid:] - INFO [Thread-12:CommitProcessor@181] - Shutting down
2017-01-24 22:12:28,779 [myid:] - INFO [Thread-12:Leader$ToBeAppliedRequestProcessor@656] - Shutting down
2017-01-24 22:12:28,779 [myid:] - INFO [Thread-12:FinalRequestProcessor@415] - shutdown of request processor complete
2017-01-24 22:12:28,779 [myid:] - INFO [Thread-12:SyncRequestProcessor@175] - Shutting down
2017-01-24 22:12:28,779 [myid:2] - INFO [ProcessThread(sid:2 cport:-1)::PrepRequestProcessor@143] - PrepRequestProcessor exited loop!
2017-01-24 22:12:28,779 [myid:2] - INFO [SyncThread:2:SyncRequestProcessor@155] - SyncRequestProcessor exited!
2017-01-24 22:12:28,779 [myid:2] - INFO [CommitProcessor:2:CommitProcessor@150] - CommitProcessor exited loop!
2017-01-24 22:12:28,782 [myid:1] - WARN [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11230:Follower@90] - Exception when following the leader
java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:392)
at org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63)
at org.apache.zookeeper.server.quorum.QuorumPacket.deserialize(QuorumPacket.java:83)
at org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:99)
at org.apache.zookeeper.server.quorum.Learner.readPacket(Learner.java:152)
at org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:86)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:851)
2017-01-24 22:12:28,784 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@224] - NIOServerCnxn factory exited run method
2017-01-24 22:12:28,782 [myid:2] - WARN [LearnerHandler-/127.0.0.1:42592:LearnerHandler@598] - ******* GOODBYE /127.0.0.1:42592 ********
2017-01-24 22:12:28,784 [myid:2] - WARN [LearnerHandler-/127.0.0.1:42592:LearnerHandler@610] - Ignoring unexpected exception
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1219)
at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:340)
at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:338)
at org.apache.zookeeper.server.quorum.LearnerHandler.shutdown(LearnerHandler.java:608)
at org.apache.zookeeper.server.quorum.LearnerHandler.run(LearnerHandler.java:601)
2017-01-24 22:12:28,785 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11230:Follower@167] - shutdown called
java.lang.Exception: shutdown Follower
at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:167)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:855)
2017-01-24 22:12:28,785 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11230:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:52356 which had sessionid 0x159d440f0ed0000
2017-01-24 22:12:28,785 [myid:] - INFO [Thread-12-SendThread(localhost:11230):ClientCnxn$SendThread@1098] - Unable to read additional data from server sessionid 0x159d440f0ed0000, likely server has closed socket, closing socket connection and attempting reconnect
2017-01-24 22:12:28,785 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11230:FollowerZooKeeperServer@139] - Shutting down
2017-01-24 22:12:28,786 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11230:ZooKeeperServer@419] - shutting down
2017-01-24 22:12:28,786 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11230:FollowerRequestProcessor@105] - Shutting down
2017-01-24 22:12:28,786 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11230:CommitProcessor@181] - Shutting down
2017-01-24 22:12:28,786 [myid:1] - INFO [FollowerRequestProcessor:1:FollowerRequestProcessor@95] - FollowerRequestProcessor exited loop!
2017-01-24 22:12:28,786 [myid:2] - WARN [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11233:QuorumPeer@867] - Unexpected exception
java.lang.InterruptedException: sleep interrupted
at java.lang.Thread.sleep(Native Method)
at org.apache.zookeeper.server.quorum.Leader.lead(Leader.java:451)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:864)
2017-01-24 22:12:28,786 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11233:Leader@491] - Shutting down
2017-01-24 22:12:28,786 [myid:2] - WARN [SendWorker:0:QuorumCnxManager$SendWorker@896] - Interrupted while waiting for message on queue
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1049)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:73)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:884)
2017-01-24 22:12:28,788 [myid:2] - WARN [SendWorker:0:QuorumCnxManager$SendWorker@906] - Send worker leaving thread
2017-01-24 22:12:28,786 [myid:2] - ERROR [localhost/127.0.0.1:11235:QuorumCnxManager$Listener@715] - Exception while listening
java.net.SocketException: Socket closed
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
at java.net.ServerSocket.implAccept(ServerSocket.java:530)
at java.net.ServerSocket.accept(ServerSocket.java:498)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$Listener.run(QuorumCnxManager.java:696)
2017-01-24 22:12:28,788 [myid:] - INFO [Thread-12:QuorumBase@318] - Shutting down leader election QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11233
2017-01-24 22:12:28,788 [myid:1] - WARN [RecvWorker:2:QuorumCnxManager$RecvWorker@980] - Connection broken for id 2, my id = 1, error =
java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:392)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:965)
2017-01-24 22:12:28,788 [myid:1] - WARN [RecvWorker:2:QuorumCnxManager$RecvWorker@983] - Interrupting SendWorker
2017-01-24 22:12:28,788 [myid:2] - WARN [RecvWorker:1:QuorumCnxManager$RecvWorker@980] - Connection broken for id 1, my id = 2, error =
java.net.SocketException: Socket closed
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at java.io.BufferedInputStream.read(BufferedInputStream.java:254)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:965)
2017-01-24 22:12:28,789 [myid:2] - WARN [RecvWorker:1:QuorumCnxManager$RecvWorker@983] - Interrupting SendWorker
2017-01-24 22:12:28,787 [myid:2] - WARN [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11233:QuorumPeer@879] - QuorumPeer main thread exited
2017-01-24 22:12:28,789 [myid:1] - WARN [SendWorker:2:QuorumCnxManager$SendWorker@896] - Interrupted while waiting for message on queue
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1049)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:73)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:884)
2017-01-24 22:12:28,787 [myid:2] - WARN [RecvWorker:0:QuorumCnxManager$RecvWorker@980] - Connection broken for id 0, my id = 2, error =
java.net.SocketException: Socket closed
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at java.io.BufferedInputStream.read(BufferedInputStream.java:254)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:965)
2017-01-24 22:12:28,790 [myid:2] - WARN [RecvWorker:0:QuorumCnxManager$RecvWorker@983] - Interrupting SendWorker
2017-01-24 22:12:28,790 [myid:1] - WARN [SendWorker:2:QuorumCnxManager$SendWorker@906] - Send worker leaving thread
2017-01-24 22:12:28,787 [myid:1] - INFO [CommitProcessor:1:CommitProcessor@150] - CommitProcessor exited loop!
2017-01-24 22:12:28,787 [myid:2] - WARN [SendWorker:1:QuorumCnxManager$SendWorker@896] - Interrupted while waiting for message on queue
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1049)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:73)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:884)
2017-01-24 22:12:28,791 [myid:2] - WARN [SendWorker:1:QuorumCnxManager$SendWorker@906] - Send worker leaving thread
2017-01-24 22:12:28,787 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11230:FinalRequestProcessor@415] - shutdown of request processor complete
2017-01-24 22:12:28,787 [myid:0] - WARN [RecvWorker:2:QuorumCnxManager$RecvWorker@980] - Connection broken for id 2, my id = 0, error =
java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:392)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:965)
2017-01-24 22:12:28,791 [myid:0] - WARN [RecvWorker:2:QuorumCnxManager$RecvWorker@983] - Interrupting SendWorker
2017-01-24 22:12:28,791 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11230:SyncRequestProcessor@175] - Shutting down
2017-01-24 22:12:28,788 [myid:] - INFO [Thread-12:QuorumBase@323] - Waiting for QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11233 to exit thread
2017-01-24 22:12:28,792 [myid:1] - INFO [SyncThread:1:SyncRequestProcessor@155] - SyncRequestProcessor exited!
2017-01-24 22:12:28,792 [myid:0] - WARN [SendWorker:2:QuorumCnxManager$SendWorker@896] - Interrupted while waiting for message on queue
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1049)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:73)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:884)
2017-01-24 22:12:28,792 [myid:0] - WARN [SendWorker:2:QuorumCnxManager$SendWorker@906] - Send worker leaving thread
2017-01-24 22:12:28,793 [myid:] - INFO [Thread-12:QuorumPeerTestBase$MainThread@81] - id = 2 tmpDir = /data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test6343054086158567637.junit.dir clientPort = 11233
2017-01-24 22:12:28,794 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11230:QuorumPeer@781] - LOOKING
2017-01-24 22:12:28,794 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11230:FileSnap@83] - Reading snapshot /data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test4772652757083079773.junit.dir/data/version-2/snapshot.200000002
2017-01-24 22:12:28,795 [myid:] - INFO [Thread-12:FourLetterWordMain@43] - connecting to 127.0.0.1 11233
2017-01-24 22:12:28,795 [myid:] - INFO [Thread-12:ClientBase@246] - server 127.0.0.1:11233 not up java.net.ConnectException: Connection refused
2017-01-24 22:12:28,795 [myid:] - INFO [Thread-47:QuorumPeerConfig@111] - Reading configuration from: /data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test6343054086158567637.junit.dir/zoo.cfg
2017-01-24 22:12:28,796 [myid:] - INFO [Thread-47:QuorumPeerConfig@374] - Defaulting to majority quorums
2017-01-24 22:12:28,796 [myid:2] - INFO [Thread-47:DatadirCleanupManager@78] - autopurge.snapRetainCount set to 3
2017-01-24 22:12:28,796 [myid:2] - INFO [Thread-47:DatadirCleanupManager@79] - autopurge.purgeInterval set to 0
2017-01-24 22:12:28,796 [myid:2] - INFO [Thread-47:DatadirCleanupManager@101] - Purge task is not scheduled.
2017-01-24 22:12:28,797 [myid:2] - WARN [Thread-47:QuorumPeerMain@129] - Unable to register log4j JMX control
javax.management.InstanceAlreadyExistsException: log4j:hiearchy=default
at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:437)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1898)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:966)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324)
at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
at org.apache.zookeeper.jmx.ManagedUtil.registerLog4jMBeans(ManagedUtil.java:53)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:127)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:116)
at org.apache.zookeeper.server.quorum.QuorumPeerTestBase$MainThread.run(QuorumPeerTestBase.java:140)
at java.lang.Thread.run(Thread.java:745)
2017-01-24 22:12:28,797 [myid:2] - INFO [Thread-47:QuorumPeerMain@132] - Starting quorum peer
2017-01-24 22:12:28,797 [myid:2] - INFO [Thread-47:NIOServerCnxnFactory@94] - binding to port 0.0.0.0/0.0.0.0:11233
2017-01-24 22:12:28,798 [myid:2] - INFO [Thread-47:QuorumPeer@1048] - minSessionTimeout set to -1
2017-01-24 22:12:28,798 [myid:2] - INFO [Thread-47:QuorumPeer@1059] - maxSessionTimeout set to -1
2017-01-24 22:12:28,798 [myid:2] - INFO [Thread-47:QuorumPeer@1279] - quorum.auth.enableSasl set to true
2017-01-24 22:12:28,798 [myid:2] - INFO [Thread-47:QuorumPeer@1264] - quorum.auth.serverRequireSasl set to false
2017-01-24 22:12:28,799 [myid:2] - INFO [Thread-47:QuorumPeer@1270] - quorum.auth.learnerRequireSasl set to true
2017-01-24 22:12:28,799 [myid:2] - INFO [Thread-47:QuorumPeer@1286] - quorum.auth.kerberos.servicePrincipal set to zkquorum/localhost
2017-01-24 22:12:28,799 [myid:2] - INFO [Thread-47:QuorumPeer@1298] - quorum.auth.server.saslLoginContext set to QuorumServer
2017-01-24 22:12:28,799 [myid:2] - INFO [Thread-47:QuorumPeer@1292] - quorum.auth.learner.saslLoginContext set to QuorumLearner
2017-01-24 22:12:28,799 [myid:2] - INFO [Thread-47:QuorumPeer@1306] - quorum.cnxn.threads.size set to 20
2017-01-24 22:12:28,799 [myid:2] - INFO [Thread-47:Login@294] - QuorumServer successfully logged in.
2017-01-24 22:12:28,800 [myid:2] - INFO [Thread-47:Login@294] - QuorumLearner successfully logged in.
2017-01-24 22:12:28,800 [myid:2] - INFO [Thread-47:QuorumPeer@540] - currentEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
2017-01-24 22:12:28,801 [myid:2] - INFO [Thread-47:QuorumPeer@555] - acceptedEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
2017-01-24 22:12:28,802 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11230:FastLeaderElection@744] - New election. My id = 1, proposed zxid=0x300000001
2017-01-24 22:12:28,802 [myid:1] - WARN [WorkerSender[myid=1]:QuorumCnxManager@559] - Cannot open channel to 2 at election address localhost/127.0.0.1:11235
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:538)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:514)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:393)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:365)
at java.lang.Thread.run(Thread.java:745)
2017-01-24 22:12:28,803 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@542] - Notification: 1 (n.leader), 0x300000001 (n.zxid), 0x4 (n.round), LOOKING (n.state), 1 (n.sid), 0x3 (n.peerEPoch), LOOKING (my state)
2017-01-24 22:12:28,803 [myid:0] - INFO [WorkerReceiver[myid=0]:FastLeaderElection@542] - Notification: 1 (n.leader), 0x300000001 (n.zxid), 0x4 (n.round), LOOKING (n.state), 1 (n.sid), 0x3 (n.peerEPoch), FOLLOWING (my state)
2017-01-24 22:12:28,803 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@542] - Notification: 2 (n.leader), 0x200000002 (n.zxid), 0x3 (n.round), FOLLOWING (n.state), 0 (n.sid), 0x2 (n.peerEPoch), LOOKING (my state)
2017-01-24 22:12:28,804 [myid:2] - INFO [Thread-48:QuorumCnxManager$Listener@691] - My election bind port: 0.0.0.0/0.0.0.0:11235
2017-01-24 22:12:28,812 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11233:QuorumPeer@781] - LOOKING
2017-01-24 22:12:28,812 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11233:FastLeaderElection@744] - New election. My id = 2, proposed zxid=0x0
2017-01-24 22:12:28,813 [myid:0] - INFO [localhost/127.0.0.1:11229:QuorumCnxManager$Listener@698] - Received connection request /127.0.0.1:51872
2017-01-24 22:12:28,821 [myid:1] - INFO [localhost/127.0.0.1:11232:QuorumCnxManager$Listener@698] - Received connection request /127.0.0.1:45220
2017-01-24 22:12:28,821 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@542] - Notification: 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)
2017-01-24 22:12:28,826 [myid:2] - INFO [QuorumConnectionThread-[myid=2]-1:SecurityUtils@68] - QuorumLearner will use DIGEST-MD5 as SASL mechanism.
2017-01-24 22:12:28,828 [myid:2] - INFO [QuorumConnectionThread-[myid=2]-2:SecurityUtils@68] - QuorumLearner will use DIGEST-MD5 as SASL mechanism.
2017-01-24 22:12:28,835 [myid:0] - INFO [QuorumConnectionThread-[myid=0]-1:SaslQuorumServerCallbackHandler@143] - Successfully authenticated learner: authenticationID=test; authorizationID=test.
2017-01-24 22:12:28,836 [myid:0] - INFO [QuorumConnectionThread-[myid=0]-1:SaslQuorumAuthServer@114] - Successfully completed the authentication using SASL. learner addr: /127.0.0.1:51872
2017-01-24 22:12:28,837 [myid:2] - INFO [QuorumConnectionThread-[myid=2]-1:SaslQuorumAuthLearner@151] - Successfully completed the authentication using SASL. server addr: localhost/127.0.0.1:11229, status: SUCCESS
2017-01-24 22:12:28,838 [myid:1] - INFO [QuorumConnectionThread-[myid=1]-1:SaslQuorumServerCallbackHandler@143] - Successfully authenticated learner: authenticationID=test; authorizationID=test.
2017-01-24 22:12:28,838 [myid:1] - INFO [QuorumConnectionThread-[myid=1]-1:SaslQuorumAuthServer@114] - Successfully completed the authentication using SASL. learner addr: /127.0.0.1:45220
2017-01-24 22:12:28,839 [myid:2] - INFO [QuorumConnectionThread-[myid=2]-2:SaslQuorumAuthLearner@151] - Successfully completed the authentication using SASL. server addr: localhost/127.0.0.1:11232, status: SUCCESS
2017-01-24 22:12:28,844 [myid:0] - INFO [WorkerReceiver[myid=0]:FastLeaderElection@542] - Notification: 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x0 (n.peerEPoch), FOLLOWING (my state)
2017-01-24 22:12:28,853 [myid:1] - WARN [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11224:QuorumCnxManager@559] - Cannot open channel to 0 at election address localhost/127.0.0.1:11223
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:538)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:579)
at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:769)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:827)
2017-01-24 22:12:28,853 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11224:FastLeaderElection@778] - Notification time out: 400
2017-01-24 22:12:28,854 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@542] - Notification: 2 (n.leader), 0x200000002 (n.zxid), 0x3 (n.round), LOOKING (n.state), 0 (n.sid), 0x2 (n.peerEPoch), LOOKING (my state)
2017-01-24 22:12:28,854 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@542] - Notification: 2 (n.leader), 0x200000002 (n.zxid), 0x3 (n.round), FOLLOWING (n.state), 0 (n.sid), 0x2 (n.peerEPoch), LOOKING (my state)
2017-01-24 22:12:28,854 [myid:0] - INFO [WorkerReceiver[myid=0]:FastLeaderElection@542] - Notification: 2 (n.leader), 0x200000002 (n.zxid), 0x3 (n.round), LOOKING (n.state), 2 (n.sid), 0x2 (n.peerEPoch), FOLLOWING (my state)
2017-01-24 22:12:28,854 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@542] - Notification: 2 (n.leader), 0x200000002 (n.zxid), 0x3 (n.round), LOOKING (n.state), 2 (n.sid), 0x2 (n.peerEPoch), LOOKING (my state)
2017-01-24 22:12:28,855 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@542] - Notification: 1 (n.leader), 0x300000001 (n.zxid), 0x4 (n.round), LOOKING (n.state), 1 (n.sid), 0x3 (n.peerEPoch), LOOKING (my state)
2017-01-24 22:12:28,855 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@542] - Notification: 2 (n.leader), 0x200000002 (n.zxid), 0x3 (n.round), FOLLOWING (n.state), 0 (n.sid), 0x2 (n.peerEPoch), LOOKING (my state)
2017-01-24 22:12:28,855 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11233:QuorumPeer@861] - LEADING
2017-01-24 22:12:28,855 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11233:ZooKeeperServer@162] - Created server with tickTime 4000 minSessionTimeout 8000 maxSessionTimeout 80000 datadir /data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test6343054086158567637.junit.dir/data/version-2 snapdir /data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test6343054086158567637.junit.dir/data/version-2
2017-01-24 22:12:28,855 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11233:Leader@356] - LEADING - LEADER ELECTION TOOK - 43
2017-01-24 22:12:28,856 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11233:FileTxnSnapLog@281] - Snapshotting: 0x0 to /data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test6343054086158567637.junit.dir/data/version-2/snapshot.0
2017-01-24 22:12:28,856 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@542] - Notification: 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)
2017-01-24 22:12:28,858 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@542] - Notification: 2 (n.leader), 0x200000002 (n.zxid), 0x3 (n.round), LOOKING (n.state), 2 (n.sid), 0x2 (n.peerEPoch), LOOKING (my state)
2017-01-24 22:12:28,859 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@542] - Notification: 1 (n.leader), 0x300000001 (n.zxid), 0x4 (n.round), LOOKING (n.state), 1 (n.sid), 0x3 (n.peerEPoch), LEADING (my state)
2017-01-24 22:12:28,859 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@542] - Notification: 1 (n.leader), 0x300000001 (n.zxid), 0x4 (n.round), LOOKING (n.state), 1 (n.sid), 0x3 (n.peerEPoch), LEADING (my state)
2017-01-24 22:12:28,859 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@542] - Notification: 2 (n.leader), 0x200000002 (n.zxid), 0x3 (n.round), LEADING (n.state), 2 (n.sid), 0x2 (n.peerEPoch), LOOKING (my state)
2017-01-24 22:12:28,859 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@542] - Notification: 2 (n.leader), 0x200000002 (n.zxid), 0x3 (n.round), LEADING (n.state), 2 (n.sid), 0x2 (n.peerEPoch), FOLLOWING (my state)
2017-01-24 22:12:28,859 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11230:QuorumPeer@849] - FOLLOWING
2017-01-24 22:12:28,860 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11230:ZooKeeperServer@162] - Created server with tickTime 4000 minSessionTimeout 8000 maxSessionTimeout 80000 datadir /data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test4772652757083079773.junit.dir/data/version-2 snapdir /data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test4772652757083079773.junit.dir/data/version-2
2017-01-24 22:12:28,860 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11230:Follower@64] - FOLLOWING - LEADER ELECTION TOOK - 66
2017-01-24 22:12:28,860 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11230:SecurityUtils@68] - QuorumLearner will use DIGEST-MD5 as SASL mechanism.
2017-01-24 22:12:28,862 [myid:2] - INFO [Thread-49:SaslQuorumServerCallbackHandler@143] - Successfully authenticated learner: authenticationID=test; authorizationID=test.
2017-01-24 22:12:28,863 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11230:SaslQuorumAuthLearner@151] - Successfully completed the authentication using SASL. server addr: localhost/127.0.0.1:11234, status: SUCCESS
2017-01-24 22:12:28,864 [myid:2] - INFO [Thread-49:SaslQuorumAuthServer@114] - Successfully completed the authentication using SASL. learner addr: /127.0.0.1:42601
2017-01-24 22:12:28,864 [myid:2] - INFO [LearnerHandler-/127.0.0.1:42601:LearnerHandler@287] - Follower sid: 1 : info : org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@2cdbf42e
2017-01-24 22:12:28,867 [myid:2] - ERROR [LearnerHandler-/127.0.0.1:42601:LearnerHandler@585] - Unexpected exception causing shutdown while sock still open
java.io.IOException: Follower is ahead of the leader
at org.apache.zookeeper.server.quorum.Leader.waitForEpochAck(Leader.java:889)
at org.apache.zookeeper.server.quorum.LearnerHandler.run(LearnerHandler.java:322)
2017-01-24 22:12:28,867 [myid:1] - WARN [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11230:Follower@90] - Exception when following the leader
java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:392)
at org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63)
at org.apache.zookeeper.server.quorum.QuorumPacket.deserialize(QuorumPacket.java:83)
at org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:99)
at org.apache.zookeeper.server.quorum.Learner.readPacket(Learner.java:152)
at org.apache.zookeeper.server.quorum.Learner.syncWithLeader(Learner.java:321)
at org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:83)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:851)
2017-01-24 22:12:28,868 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11230:Follower@167] - shutdown called
java.lang.Exception: shutdown Follower
at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:167)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:855)
2017-01-24 22:12:28,868 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11230:FollowerZooKeeperServer@139] - Shutting down
2017-01-24 22:12:28,868 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11230:ZooKeeperServer@419] - shutting down
2017-01-24 22:12:28,868 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11230:QuorumPeer@781] - LOOKING
2017-01-24 22:12:28,869 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11230:FileSnap@83] - Reading snapshot /data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test4772652757083079773.junit.dir/data/version-2/snapshot.200000002
2017-01-24 22:12:28,870 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11230:FastLeaderElection@744] - New election. My id = 1, proposed zxid=0x300000001
2017-01-24 22:12:28,871 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@542] - Notification: 1 (n.leader), 0x300000001 (n.zxid), 0x4 (n.round), LOOKING (n.state), 1 (n.sid), 0x3 (n.peerEPoch), LOOKING (my state)
2017-01-24 22:12:28,871 [myid:0] - INFO [WorkerReceiver[myid=0]:FastLeaderElection@542] - Notification: 1 (n.leader), 0x300000001 (n.zxid), 0x4 (n.round), LOOKING (n.state), 1 (n.sid), 0x3 (n.peerEPoch), FOLLOWING (my state)
2017-01-24 22:12:28,871 [myid:2] - WARN [LearnerHandler-/127.0.0.1:42601:LearnerHandler@598] - ******* GOODBYE /127.0.0.1:42601 ********
2017-01-24 22:12:28,872 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@542] - Notification: 2 (n.leader), 0x200000002 (n.zxid), 0x3 (n.round), FOLLOWING (n.state), 0 (n.sid), 0x2 (n.peerEPoch), LOOKING (my state)
2017-01-24 22:12:28,871 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@542] - Notification: 1 (n.leader), 0x300000001 (n.zxid), 0x4 (n.round), LOOKING (n.state), 1 (n.sid), 0x3 (n.peerEPoch), LEADING (my state)
2017-01-24 22:12:28,872 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@542] - Notification: 2 (n.leader), 0x200000002 (n.zxid), 0x3 (n.round), LEADING (n.state), 2 (n.sid), 0x2 (n.peerEPoch), LOOKING (my state)
2017-01-24 22:12:28,873 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11230:QuorumPeer@849] - FOLLOWING
2017-01-24 22:12:28,873 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11230:ZooKeeperServer@162] - Created server with tickTime 4000 minSessionTimeout 8000 maxSessionTimeout 80000 datadir /data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test4772652757083079773.junit.dir/data/version-2 snapdir /data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test4772652757083079773.junit.dir/data/version-2
2017-01-24 22:12:28,873 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11230:Follower@64] - FOLLOWING - LEADER ELECTION TOOK - 5
2017-01-24 22:12:28,874 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11230:SecurityUtils@68] - QuorumLearner will use DIGEST-MD5 as SASL mechanism.
2017-01-24 22:12:28,875 [myid:2] - INFO [Thread-49:SaslQuorumServerCallbackHandler@143] - Successfully authenticated learner: authenticationID=test; authorizationID=test.
2017-01-24 22:12:28,876 [myid:2] - INFO [Thread-49:SaslQuorumAuthServer@114] - Successfully completed the authentication using SASL. learner addr: /127.0.0.1:42602
2017-01-24 22:12:28,876 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11230:SaslQuorumAuthLearner@151] - Successfully completed the authentication using SASL. server addr: localhost/127.0.0.1:11234, status: SUCCESS
2017-01-24 22:12:28,877 [myid:2] - INFO [LearnerHandler-/127.0.0.1:42602:LearnerHandler@287] - Follower sid: 1 : info : org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@2cdbf42e
2017-01-24 22:12:28,976 [myid:1] - INFO [localhost/127.0.0.1:11232:QuorumCnxManager$Listener@728] - Leaving listener
2017-01-24 22:12:29,045 [myid:] - INFO [Thread-12:FourLetterWordMain@43] - connecting to 127.0.0.1 11233
2017-01-24 22:12:29,046 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45833
2017-01-24 22:12:29,046 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:45833
2017-01-24 22:12:29,047 [myid:2] - INFO [Thread-50:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45833 (no session established for client)
2017-01-24 22:12:29,184 [myid:] - WARN [Thread-12-SendThread(localhost:11233):ClientCnxn$SendThread@957] - SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: '/data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test2999887027410032136.junit.dir/jaas.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.
2017-01-24 22:12:29,184 [myid:] - INFO [Thread-12-SendThread(localhost:11233):ClientCnxn$SendThread@975] - Opening socket connection to server localhost/127.0.0.1:11233
2017-01-24 22:12:29,185 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45834
2017-01-24 22:12:29,185 [myid:] - INFO [Thread-12-SendThread(localhost:11233):ClientCnxn$SendThread@852] - Socket connection established, initiating session, client: /127.0.0.1:45834, server: localhost/127.0.0.1:11233
2017-01-24 22:12:29,185 [myid:2] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@354] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2017-01-24 22:12:29,185 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45834 (no session established for client)
2017-01-24 22:12:29,185 [myid:] - INFO [Thread-12-SendThread(localhost:11233):ClientCnxn$SendThread@1098] - Unable to read additional data from server sessionid 0x159d440f0ed0000, likely server has closed socket, closing socket connection and attempting reconnect
2017-01-24 22:12:29,254 [myid:1] - WARN [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11224:QuorumCnxManager@559] - Cannot open channel to 0 at election address localhost/127.0.0.1:11223
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:538)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:579)
at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:769)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:827)
2017-01-24 22:12:29,254 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11224:FastLeaderElection@778] - Notification time out: 800
2017-01-24 22:12:29,297 [myid:] - INFO [Thread-12:FourLetterWordMain@43] - connecting to 127.0.0.1 11233
2017-01-24 22:12:29,298 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45836
2017-01-24 22:12:29,298 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:45836
2017-01-24 22:12:29,299 [myid:2] - INFO [Thread-51:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45836 (no session established for client)
2017-01-24 22:12:29,363 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11227:SaslQuorumAuthLearner@79] - Skipping SASL authentication as quorum.auth.learnerRequireSasl=false
2017-01-24 22:12:29,365 [myid:2] - INFO [LearnerHandler-/127.0.0.1:42607:LearnerHandler@287] - Follower sid: 0 : info : org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@76415e70
2017-01-24 22:12:29,366 [myid:2] - ERROR [LearnerHandler-/127.0.0.1:42607:LearnerHandler@585] - Unexpected exception causing shutdown while sock still open
java.io.IOException: Follower is ahead of the leader
at org.apache.zookeeper.server.quorum.Leader.waitForEpochAck(Leader.java:889)
at org.apache.zookeeper.server.quorum.LearnerHandler.run(LearnerHandler.java:322)
2017-01-24 22:12:29,366 [myid:2] - WARN [LearnerHandler-/127.0.0.1:42607:LearnerHandler@598] - ******* GOODBYE /127.0.0.1:42607 ********
2017-01-24 22:12:29,366 [myid:0] - WARN [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11227:Follower@90] - Exception when following the leader
java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:392)
at org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63)
at org.apache.zookeeper.server.quorum.QuorumPacket.deserialize(QuorumPacket.java:83)
at org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:99)
at org.apache.zookeeper.server.quorum.Learner.readPacket(Learner.java:152)
at org.apache.zookeeper.server.quorum.Learner.syncWithLeader(Learner.java:321)
at org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:83)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:851)
2017-01-24 22:12:29,367 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11227:Follower@167] - shutdown called
java.lang.Exception: shutdown Follower
at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:167)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:855)
2017-01-24 22:12:29,367 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11227:FollowerZooKeeperServer@139] - Shutting down
2017-01-24 22:12:29,367 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11227:ZooKeeperServer@419] - shutting down
2017-01-24 22:12:29,367 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11227:QuorumPeer@781] - LOOKING
2017-01-24 22:12:29,368 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11227:FileSnap@83] - Reading snapshot /data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test6261823493417515862.junit.dir/data/version-2/snapshot.100000003
2017-01-24 22:12:29,369 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11227:FastLeaderElection@744] - New election. My id = 0, proposed zxid=0x200000002
2017-01-24 22:12:29,370 [myid:0] - INFO [WorkerReceiver[myid=0]:FastLeaderElection@542] - Notification: 0 (n.leader), 0x200000002 (n.zxid), 0x4 (n.round), LOOKING (n.state), 0 (n.sid), 0x2 (n.peerEPoch), LOOKING (my state)
2017-01-24 22:12:29,370 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@542] - Notification: 0 (n.leader), 0x200000002 (n.zxid), 0x4 (n.round), LOOKING (n.state), 0 (n.sid), 0x2 (n.peerEPoch), LEADING (my state)
2017-01-24 22:12:29,370 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@542] - Notification: 0 (n.leader), 0x200000002 (n.zxid), 0x4 (n.round), LOOKING (n.state), 0 (n.sid), 0x2 (n.peerEPoch), FOLLOWING (my state)
2017-01-24 22:12:29,371 [myid:0] - INFO [WorkerReceiver[myid=0]:FastLeaderElection@542] - Notification: 2 (n.leader), 0x200000002 (n.zxid), 0x3 (n.round), LEADING (n.state), 2 (n.sid), 0x2 (n.peerEPoch), LOOKING (my state)
2017-01-24 22:12:29,371 [myid:0] - INFO [WorkerReceiver[myid=0]:FastLeaderElection@542] - Notification: 2 (n.leader), 0x200000002 (n.zxid), 0x3 (n.round), FOLLOWING (n.state), 1 (n.sid), 0x2 (n.peerEPoch), LOOKING (my state)
2017-01-24 22:12:29,371 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11227:QuorumPeer@849] - FOLLOWING
2017-01-24 22:12:29,371 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11227:ZooKeeperServer@162] - Created server with tickTime 4000 minSessionTimeout 8000 maxSessionTimeout 80000 datadir /data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test6261823493417515862.junit.dir/data/version-2 snapdir /data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test6261823493417515862.junit.dir/data/version-2
2017-01-24 22:12:29,371 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11227:Follower@64] - FOLLOWING - LEADER ELECTION TOOK - 3
2017-01-24 22:12:29,372 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11227:SaslQuorumAuthLearner@79] - Skipping SASL authentication as quorum.auth.learnerRequireSasl=false
2017-01-24 22:12:29,372 [myid:2] - INFO [LearnerHandler-/127.0.0.1:42608:LearnerHandler@287] - Follower sid: 0 : info : org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@76415e70
2017-01-24 22:12:29,386 [myid:] - WARN [Thread-12-SendThread(localhost:11227):ClientCnxn$SendThread@957] - SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: '/data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test2999887027410032136.junit.dir/jaas.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.
2017-01-24 22:12:29,386 [myid:] - INFO [Thread-12-SendThread(localhost:11227):ClientCnxn$SendThread@975] - Opening socket connection to server localhost/127.0.0.1:11227
2017-01-24 22:12:29,386 [myid:0] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11227:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:42869
2017-01-24 22:12:29,386 [myid:] - INFO [Thread-12-SendThread(localhost:11227):ClientCnxn$SendThread@852] - Socket connection established, initiating session, client: /127.0.0.1:42869, server: localhost/127.0.0.1:11227
2017-01-24 22:12:29,387 [myid:0] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11227:NIOServerCnxn@354] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2017-01-24 22:12:29,387 [myid:0] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11227:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:42869 (no session established for client)
2017-01-24 22:12:29,387 [myid:] - INFO [Thread-12-SendThread(localhost:11227):ClientCnxn$SendThread@1098] - Unable to read additional data from server sessionid 0x159d440f0ed0000, likely server has closed socket, closing socket connection and attempting reconnect
2017-01-24 22:12:29,549 [myid:] - INFO [Thread-12:FourLetterWordMain@43] - connecting to 127.0.0.1 11233
2017-01-24 22:12:29,550 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45840
2017-01-24 22:12:29,550 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:45840
2017-01-24 22:12:29,551 [myid:2] - INFO [Thread-52:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45840 (no session established for client)
2017-01-24 22:12:29,788 [myid:2] - INFO [localhost/127.0.0.1:11235:QuorumCnxManager$Listener@728] - Leaving listener
2017-01-24 22:12:29,801 [myid:] - INFO [Thread-12:FourLetterWordMain@43] - connecting to 127.0.0.1 11233
2017-01-24 22:12:29,801 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45841
2017-01-24 22:12:29,802 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:45841
2017-01-24 22:12:29,802 [myid:2] - INFO [Thread-53:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45841 (no session established for client)
2017-01-24 22:12:30,052 [myid:] - INFO [Thread-12:FourLetterWordMain@43] - connecting to 127.0.0.1 11233
2017-01-24 22:12:30,053 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45842
2017-01-24 22:12:30,053 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:45842
2017-01-24 22:12:30,054 [myid:2] - INFO [Thread-54:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45842 (no session established for client)
2017-01-24 22:12:30,055 [myid:1] - WARN [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11224:QuorumCnxManager@559] - Cannot open channel to 0 at election address localhost/127.0.0.1:11223
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:538)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:579)
at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:769)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:827)
2017-01-24 22:12:30,055 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11224:FastLeaderElection@778] - Notification time out: 1600
2017-01-24 22:12:30,304 [myid:] - INFO [Thread-12:FourLetterWordMain@43] - connecting to 127.0.0.1 11233
2017-01-24 22:12:30,305 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45844
2017-01-24 22:12:30,305 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:45844
2017-01-24 22:12:30,306 [myid:2] - INFO [Thread-55:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45844 (no session established for client)
2017-01-24 22:12:30,502 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection$Messenger$WorkerReceiver@340] - WorkerReceiver is down
2017-01-24 22:12:30,502 [myid:1] - INFO [WorkerSender[myid=1]:FastLeaderElection$Messenger$WorkerSender@370] - WorkerSender is down
2017-01-24 22:12:30,556 [myid:] - INFO [Thread-12:FourLetterWordMain@43] - connecting to 127.0.0.1 11233
2017-01-24 22:12:30,557 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45845
2017-01-24 22:12:30,557 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:45845
2017-01-24 22:12:30,557 [myid:2] - INFO [Thread-56:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45845 (no session established for client)
2017-01-24 22:12:30,808 [myid:] - INFO [Thread-12:FourLetterWordMain@43] - connecting to 127.0.0.1 11233
2017-01-24 22:12:30,808 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45846
2017-01-24 22:12:30,809 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:45846
2017-01-24 22:12:30,809 [myid:2] - INFO [Thread-57:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45846 (no session established for client)
2017-01-24 22:12:30,993 [myid:2] - INFO [WorkerSender[myid=2]:FastLeaderElection$Messenger$WorkerSender@370] - WorkerSender is down
2017-01-24 22:12:31,059 [myid:] - INFO [Thread-12:FourLetterWordMain@43] - connecting to 127.0.0.1 11233
2017-01-24 22:12:31,060 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45847
2017-01-24 22:12:31,060 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:45847
2017-01-24 22:12:31,061 [myid:2] - INFO [Thread-58:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45847 (no session established for client)
2017-01-24 22:12:31,189 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@340] - WorkerReceiver is down
2017-01-24 22:12:31,311 [myid:] - INFO [Thread-12:FourLetterWordMain@43] - connecting to 127.0.0.1 11233
2017-01-24 22:12:31,312 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45848
2017-01-24 22:12:31,312 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:45848
2017-01-24 22:12:31,313 [myid:2] - INFO [Thread-59:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45848 (no session established for client)
2017-01-24 22:12:31,365 [myid:] - WARN [Thread-12-SendThread(localhost:11230):ClientCnxn$SendThread@957] - SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: '/data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test2999887027410032136.junit.dir/jaas.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.
2017-01-24 22:12:31,365 [myid:] - INFO [Thread-12-SendThread(localhost:11230):ClientCnxn$SendThread@975] - Opening socket connection to server localhost/127.0.0.1:11230
2017-01-24 22:12:31,365 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11230:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:52380
2017-01-24 22:12:31,365 [myid:] - INFO [Thread-12-SendThread(localhost:11230):ClientCnxn$SendThread@852] - Socket connection established, initiating session, client: /127.0.0.1:52380, server: localhost/127.0.0.1:11230
2017-01-24 22:12:31,366 [myid:1] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11230:NIOServerCnxn@354] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2017-01-24 22:12:31,366 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11230:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:52380 (no session established for client)
2017-01-24 22:12:31,366 [myid:] - INFO [Thread-12-SendThread(localhost:11230):ClientCnxn$SendThread@1098] - Unable to read additional data from server sessionid 0x159d440f0ed0000, likely server has closed socket, closing socket connection and attempting reconnect
2017-01-24 22:12:31,563 [myid:] - INFO [Thread-12:FourLetterWordMain@43] - connecting to 127.0.0.1 11233
2017-01-24 22:12:31,563 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45850
2017-01-24 22:12:31,564 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:45850
2017-01-24 22:12:31,564 [myid:2] - INFO [Thread-60:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45850 (no session established for client)
2017-01-24 22:12:31,656 [myid:1] - WARN [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11224:QuorumCnxManager@559] - Cannot open channel to 0 at election address localhost/127.0.0.1:11223
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:538)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:579)
at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:769)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:827)
2017-01-24 22:12:31,656 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11224:FastLeaderElection@778] - Notification time out: 3200
2017-01-24 22:12:31,815 [myid:] - INFO [Thread-12:FourLetterWordMain@43] - connecting to 127.0.0.1 11233
2017-01-24 22:12:31,815 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45852
2017-01-24 22:12:31,815 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:45852
2017-01-24 22:12:31,816 [myid:2] - INFO [Thread-61:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45852 (no session established for client)
2017-01-24 22:12:32,000 [myid:2] - INFO [SessionTracker:SessionTrackerImpl@162] - SessionTrackerImpl exited loop!
2017-01-24 22:12:32,000 [myid:1] - INFO [SessionTracker:SessionTrackerImpl@162] - SessionTrackerImpl exited loop!
2017-01-24 22:12:32,046 [myid:] - WARN [Thread-12-SendThread(localhost:11233):ClientCnxn$SendThread@957] - SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: '/data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test2999887027410032136.junit.dir/jaas.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.
2017-01-24 22:12:32,046 [myid:] - INFO [Thread-12-SendThread(localhost:11233):ClientCnxn$SendThread@975] - Opening socket connection to server localhost/127.0.0.1:11233
2017-01-24 22:12:32,046 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45853
2017-01-24 22:12:32,046 [myid:] - INFO [Thread-12-SendThread(localhost:11233):ClientCnxn$SendThread@852] - Socket connection established, initiating session, client: /127.0.0.1:45853, server: localhost/127.0.0.1:11233
2017-01-24 22:12:32,047 [myid:2] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@354] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2017-01-24 22:12:32,047 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45853 (no session established for client)
2017-01-24 22:12:32,047 [myid:] - INFO [Thread-12-SendThread(localhost:11233):ClientCnxn$SendThread@1098] - Unable to read additional data from server sessionid 0x159d440f0ed0000, likely server has closed socket, closing socket connection and attempting reconnect
2017-01-24 22:12:32,066 [myid:] - INFO [Thread-12:FourLetterWordMain@43] - connecting to 127.0.0.1 11233
2017-01-24 22:12:32,067 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45854
2017-01-24 22:12:32,067 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:45854
2017-01-24 22:12:32,068 [myid:2] - INFO [Thread-62:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45854 (no session established for client)
2017-01-24 22:12:32,318 [myid:] - INFO [Thread-12:FourLetterWordMain@43] - connecting to 127.0.0.1 11233
2017-01-24 22:12:32,319 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45855
2017-01-24 22:12:32,319 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:45855
2017-01-24 22:12:32,320 [myid:2] - INFO [Thread-63:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45855 (no session established for client)
2017-01-24 22:12:32,570 [myid:] - INFO [Thread-12:FourLetterWordMain@43] - connecting to 127.0.0.1 11233
2017-01-24 22:12:32,570 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45856
2017-01-24 22:12:32,571 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:45856
2017-01-24 22:12:32,571 [myid:2] - INFO [Thread-64:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45856 (no session established for client)
2017-01-24 22:12:32,710 [myid:] - WARN [Thread-12-SendThread(localhost:11227):ClientCnxn$SendThread@957] - SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: '/data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test2999887027410032136.junit.dir/jaas.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.
2017-01-24 22:12:32,711 [myid:] - INFO [Thread-12-SendThread(localhost:11227):ClientCnxn$SendThread@975] - Opening socket connection to server localhost/127.0.0.1:11227
2017-01-24 22:12:32,711 [myid:0] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11227:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:42887
2017-01-24 22:12:32,711 [myid:] - INFO [Thread-12-SendThread(localhost:11227):ClientCnxn$SendThread@852] - Socket connection established, initiating session, client: /127.0.0.1:42887, server: localhost/127.0.0.1:11227
2017-01-24 22:12:32,711 [myid:0] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11227:NIOServerCnxn@354] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2017-01-24 22:12:32,712 [myid:0] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11227:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:42887 (no session established for client)
2017-01-24 22:12:32,712 [myid:] - INFO [Thread-12-SendThread(localhost:11227):ClientCnxn$SendThread@1098] - Unable to read additional data from server sessionid 0x159d440f0ed0000, likely server has closed socket, closing socket connection and attempting reconnect
2017-01-24 22:12:32,822 [myid:] - INFO [Thread-12:FourLetterWordMain@43] - connecting to 127.0.0.1 11233
2017-01-24 22:12:32,822 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45858
2017-01-24 22:12:32,822 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:45858
2017-01-24 22:12:32,823 [myid:2] - INFO [Thread-65:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45858 (no session established for client)
2017-01-24 22:12:33,073 [myid:] - INFO [Thread-12:FourLetterWordMain@43] - connecting to 127.0.0.1 11233
2017-01-24 22:12:33,074 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45859
2017-01-24 22:12:33,074 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:45859
2017-01-24 22:12:33,075 [myid:2] - INFO [Thread-66:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45859 (no session established for client)
2017-01-24 22:12:33,325 [myid:] - INFO [Thread-12:FourLetterWordMain@43] - connecting to 127.0.0.1 11233
2017-01-24 22:12:33,326 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45860
2017-01-24 22:12:33,326 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:45860
2017-01-24 22:12:33,326 [myid:2] - INFO [Thread-67:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45860 (no session established for client)
2017-01-24 22:12:33,577 [myid:] - INFO [Thread-12:FourLetterWordMain@43] - connecting to 127.0.0.1 11233
2017-01-24 22:12:33,577 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45861
2017-01-24 22:12:33,578 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:45861
2017-01-24 22:12:33,578 [myid:2] - INFO [Thread-68:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45861 (no session established for client)
2017-01-24 22:12:33,828 [myid:] - INFO [Thread-12:FourLetterWordMain@43] - connecting to 127.0.0.1 11233
2017-01-24 22:12:33,829 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45862
2017-01-24 22:12:33,829 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:45862
2017-01-24 22:12:33,830 [myid:2] - INFO [Thread-69:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45862 (no session established for client)
2017-01-24 22:12:34,080 [myid:] - INFO [Thread-12:FourLetterWordMain@43] - connecting to 127.0.0.1 11233
2017-01-24 22:12:34,081 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45863
2017-01-24 22:12:34,081 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:45863
2017-01-24 22:12:34,082 [myid:2] - INFO [Thread-70:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45863 (no session established for client)
2017-01-24 22:12:34,332 [myid:] - INFO [Thread-12:FourLetterWordMain@43] - connecting to 127.0.0.1 11233
2017-01-24 22:12:34,332 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45864
2017-01-24 22:12:34,333 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:45864
2017-01-24 22:12:34,333 [myid:2] - INFO [Thread-71:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45864 (no session established for client)
2017-01-24 22:12:34,421 [myid:] - WARN [Thread-12-SendThread(localhost:11230):ClientCnxn$SendThread@957] - SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: '/data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test2999887027410032136.junit.dir/jaas.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.
2017-01-24 22:12:34,421 [myid:] - INFO [Thread-12-SendThread(localhost:11230):ClientCnxn$SendThread@975] - Opening socket connection to server localhost/127.0.0.1:11230
2017-01-24 22:12:34,421 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11230:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:52396
2017-01-24 22:12:34,421 [myid:] - INFO [Thread-12-SendThread(localhost:11230):ClientCnxn$SendThread@852] - Socket connection established, initiating session, client: /127.0.0.1:52396, server: localhost/127.0.0.1:11230
2017-01-24 22:12:34,422 [myid:1] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11230:NIOServerCnxn@354] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2017-01-24 22:12:34,422 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11230:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:52396 (no session established for client)
2017-01-24 22:12:34,422 [myid:] - INFO [Thread-12-SendThread(localhost:11230):ClientCnxn$SendThread@1098] - Unable to read additional data from server sessionid 0x159d440f0ed0000, likely server has closed socket, closing socket connection and attempting reconnect
2017-01-24 22:12:34,584 [myid:] - INFO [Thread-12:FourLetterWordMain@43] - connecting to 127.0.0.1 11233
2017-01-24 22:12:34,584 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45866
2017-01-24 22:12:34,584 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:45866
2017-01-24 22:12:34,585 [myid:2] - INFO [Thread-72:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45866 (no session established for client)
2017-01-24 22:12:34,835 [myid:] - INFO [Thread-12:FourLetterWordMain@43] - connecting to 127.0.0.1 11233
2017-01-24 22:12:34,836 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45867
2017-01-24 22:12:34,836 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:45867
2017-01-24 22:12:34,837 [myid:2] - INFO [Thread-73:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45867 (no session established for client)
2017-01-24 22:12:34,857 [myid:1] - WARN [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11224:QuorumCnxManager@559] - Cannot open channel to 0 at election address localhost/127.0.0.1:11223
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:538)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:579)
at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:769)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:827)
2017-01-24 22:12:34,857 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11224:FastLeaderElection@778] - Notification time out: 6400
2017-01-24 22:12:35,087 [myid:] - INFO [Thread-12:FourLetterWordMain@43] - connecting to 127.0.0.1 11233
2017-01-24 22:12:35,088 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45869
2017-01-24 22:12:35,088 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:45869
2017-01-24 22:12:35,088 [myid:2] - INFO [Thread-74:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45869 (no session established for client)
2017-01-24 22:12:35,339 [myid:] - INFO [Thread-12:FourLetterWordMain@43] - connecting to 127.0.0.1 11233
2017-01-24 22:12:35,339 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45870
2017-01-24 22:12:35,340 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:45870
2017-01-24 22:12:35,340 [myid:2] - INFO [Thread-75:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45870 (no session established for client)
2017-01-24 22:12:35,424 [myid:] - WARN [Thread-12-SendThread(localhost:11233):ClientCnxn$SendThread@957] - SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: '/data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test2999887027410032136.junit.dir/jaas.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.
2017-01-24 22:12:35,425 [myid:] - INFO [Thread-12-SendThread(localhost:11233):ClientCnxn$SendThread@975] - Opening socket connection to server localhost/127.0.0.1:11233
2017-01-24 22:12:35,425 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45871
2017-01-24 22:12:35,425 [myid:] - INFO [Thread-12-SendThread(localhost:11233):ClientCnxn$SendThread@852] - Socket connection established, initiating session, client: /127.0.0.1:45871, server: localhost/127.0.0.1:11233
2017-01-24 22:12:35,425 [myid:2] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@354] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2017-01-24 22:12:35,426 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45871 (no session established for client)
2017-01-24 22:12:35,426 [myid:] - INFO [Thread-12-SendThread(localhost:11233):ClientCnxn$SendThread@1098] - Unable to read additional data from server sessionid 0x159d440f0ed0000, likely server has closed socket, closing socket connection and attempting reconnect
2017-01-24 22:12:35,590 [myid:] - INFO [Thread-12:FourLetterWordMain@43] - connecting to 127.0.0.1 11233
2017-01-24 22:12:35,591 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45872
2017-01-24 22:12:35,591 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:45872
2017-01-24 22:12:35,592 [myid:2] - INFO [Thread-76:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45872 (no session established for client)
2017-01-24 22:12:35,842 [myid:] - INFO [Thread-12:FourLetterWordMain@43] - connecting to 127.0.0.1 11233
2017-01-24 22:12:35,843 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45873
2017-01-24 22:12:35,843 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:45873
2017-01-24 22:12:35,844 [myid:2] - INFO [Thread-77:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45873 (no session established for client)
2017-01-24 22:12:36,094 [myid:] - INFO [Thread-12:FourLetterWordMain@43] - connecting to 127.0.0.1 11233
2017-01-24 22:12:36,095 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45874
2017-01-24 22:12:36,095 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:45874
2017-01-24 22:12:36,095 [myid:2] - INFO [Thread-78:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45874 (no session established for client)
2017-01-24 22:12:36,205 [myid:] - WARN [Thread-12-SendThread(localhost:11227):ClientCnxn$SendThread@957] - SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: '/data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test2999887027410032136.junit.dir/jaas.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.
2017-01-24 22:12:36,206 [myid:] - INFO [Thread-12-SendThread(localhost:11227):ClientCnxn$SendThread@975] - Opening socket connection to server localhost/127.0.0.1:11227
2017-01-24 22:12:36,206 [myid:0] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11227:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:42905
2017-01-24 22:12:36,206 [myid:] - INFO [Thread-12-SendThread(localhost:11227):ClientCnxn$SendThread@852] - Socket connection established, initiating session, client: /127.0.0.1:42905, server: localhost/127.0.0.1:11227
2017-01-24 22:12:36,207 [myid:0] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11227:NIOServerCnxn@354] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2017-01-24 22:12:36,207 [myid:0] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11227:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:42905 (no session established for client)
2017-01-24 22:12:36,207 [myid:] - INFO [Thread-12-SendThread(localhost:11227):ClientCnxn$SendThread@1098] - Unable to read additional data from server sessionid 0x159d440f0ed0000, likely server has closed socket, closing socket connection and attempting reconnect
2017-01-24 22:12:36,346 [myid:] - INFO [Thread-12:FourLetterWordMain@43] - connecting to 127.0.0.1 11233
2017-01-24 22:12:36,346 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45876
2017-01-24 22:12:36,347 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:45876
2017-01-24 22:12:36,347 [myid:2] - INFO [Thread-79:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45876 (no session established for client)
2017-01-24 22:12:36,597 [myid:] - INFO [Thread-12:FourLetterWordMain@43] - connecting to 127.0.0.1 11233
2017-01-24 22:12:36,598 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45877
2017-01-24 22:12:36,598 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:45877
2017-01-24 22:12:36,599 [myid:2] - INFO [Thread-80:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45877 (no session established for client)
2017-01-24 22:12:36,849 [myid:] - INFO [Thread-12:FourLetterWordMain@43] - connecting to 127.0.0.1 11233
2017-01-24 22:12:36,850 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45878
2017-01-24 22:12:36,850 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:45878
2017-01-24 22:12:36,850 [myid:2] - INFO [Thread-81:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45878 (no session established for client)
2017-01-24 22:12:37,101 [myid:] - INFO [Thread-12:FourLetterWordMain@43] - connecting to 127.0.0.1 11233
2017-01-24 22:12:37,101 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45879
2017-01-24 22:12:37,102 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:45879
2017-01-24 22:12:37,102 [myid:2] - INFO [Thread-82:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45879 (no session established for client)
2017-01-24 22:12:37,352 [myid:] - INFO [Thread-12:FourLetterWordMain@43] - connecting to 127.0.0.1 11233
2017-01-24 22:12:37,353 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45880
2017-01-24 22:12:37,353 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:45880
2017-01-24 22:12:37,354 [myid:2] - INFO [Thread-83:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45880 (no session established for client)
2017-01-24 22:12:37,604 [myid:] - INFO [Thread-12:FourLetterWordMain@43] - connecting to 127.0.0.1 11233
2017-01-24 22:12:37,605 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45881
2017-01-24 22:12:37,605 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:45881
2017-01-24 22:12:37,606 [myid:2] - INFO [Thread-84:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45881 (no session established for client)
2017-01-24 22:12:37,856 [myid:] - INFO [Thread-12:FourLetterWordMain@43] - connecting to 127.0.0.1 11233
2017-01-24 22:12:37,857 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45882
2017-01-24 22:12:37,857 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:45882
2017-01-24 22:12:37,857 [myid:2] - INFO [Thread-85:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45882 (no session established for client)
2017-01-24 22:12:38,108 [myid:] - INFO [Thread-12:FourLetterWordMain@43] - connecting to 127.0.0.1 11233
2017-01-24 22:12:38,108 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45883
2017-01-24 22:12:38,109 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:45883
2017-01-24 22:12:38,109 [myid:2] - INFO [Thread-86:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45883 (no session established for client)
2017-01-24 22:12:38,282 [myid:] - WARN [Thread-12-SendThread(localhost:11230):ClientCnxn$SendThread@957] - SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: '/data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test2999887027410032136.junit.dir/jaas.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.
2017-01-24 22:12:38,282 [myid:] - INFO [Thread-12-SendThread(localhost:11230):ClientCnxn$SendThread@975] - Opening socket connection to server localhost/127.0.0.1:11230
2017-01-24 22:12:38,282 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11230:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:52415
2017-01-24 22:12:38,282 [myid:] - INFO [Thread-12-SendThread(localhost:11230):ClientCnxn$SendThread@852] - Socket connection established, initiating session, client: /127.0.0.1:52415, server: localhost/127.0.0.1:11230
2017-01-24 22:12:38,283 [myid:1] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11230:NIOServerCnxn@354] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2017-01-24 22:12:38,283 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11230:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:52415 (no session established for client)
2017-01-24 22:12:38,283 [myid:] - INFO [Thread-12-SendThread(localhost:11230):ClientCnxn$SendThread@1098] - Unable to read additional data from server sessionid 0x159d440f0ed0000, likely server has closed socket, closing socket connection and attempting reconnect
2017-01-24 22:12:38,360 [myid:] - INFO [Thread-12:FourLetterWordMain@43] - connecting to 127.0.0.1 11233
2017-01-24 22:12:38,360 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45885
2017-01-24 22:12:38,360 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:45885
2017-01-24 22:12:38,361 [myid:2] - INFO [Thread-87:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45885 (no session established for client)
2017-01-24 22:12:38,611 [myid:] - INFO [Thread-12:FourLetterWordMain@43] - connecting to 127.0.0.1 11233
2017-01-24 22:12:38,612 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45886
2017-01-24 22:12:38,612 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:45886
2017-01-24 22:12:38,613 [myid:2] - INFO [Thread-88:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45886 (no session established for client)
2017-01-24 22:12:38,863 [myid:] - INFO [Thread-12:FourLetterWordMain@43] - connecting to 127.0.0.1 11233
2017-01-24 22:12:38,864 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45887
2017-01-24 22:12:38,864 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:45887
2017-01-24 22:12:38,864 [myid:2] - INFO [Thread-89:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45887 (no session established for client)
2017-01-24 22:12:39,115 [myid:] - INFO [Thread-12:FourLetterWordMain@43] - connecting to 127.0.0.1 11233
2017-01-24 22:12:39,115 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45888
2017-01-24 22:12:39,116 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:45888
2017-01-24 22:12:39,116 [myid:2] - INFO [Thread-90:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45888 (no session established for client)
2017-01-24 22:12:39,296 [myid:] - WARN [Thread-12-SendThread(localhost:11233):ClientCnxn$SendThread@957] - SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: '/data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test2999887027410032136.junit.dir/jaas.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.
2017-01-24 22:12:39,297 [myid:] - INFO [Thread-12-SendThread(localhost:11233):ClientCnxn$SendThread@975] - Opening socket connection to server localhost/127.0.0.1:11233
2017-01-24 22:12:39,297 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45889
2017-01-24 22:12:39,297 [myid:] - INFO [Thread-12-SendThread(localhost:11233):ClientCnxn$SendThread@852] - Socket connection established, initiating session, client: /127.0.0.1:45889, server: localhost/127.0.0.1:11233
2017-01-24 22:12:39,298 [myid:2] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@354] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2017-01-24 22:12:39,298 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45889 (no session established for client)
2017-01-24 22:12:39,298 [myid:] - INFO [Thread-12-SendThread(localhost:11233):ClientCnxn$SendThread@1098] - Unable to read additional data from server sessionid 0x159d440f0ed0000, likely server has closed socket, closing socket connection and attempting reconnect
2017-01-24 22:12:39,367 [myid:] - INFO [Thread-12:FourLetterWordMain@43] - connecting to 127.0.0.1 11233
2017-01-24 22:12:39,367 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45890
2017-01-24 22:12:39,367 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:45890
2017-01-24 22:12:39,368 [myid:2] - INFO [Thread-91:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45890 (no session established for client)
2017-01-24 22:12:39,618 [myid:] - INFO [Thread-12:FourLetterWordMain@43] - connecting to 127.0.0.1 11233
2017-01-24 22:12:39,619 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45891
2017-01-24 22:12:39,619 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:45891
2017-01-24 22:12:39,620 [myid:2] - INFO [Thread-92:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45891 (no session established for client)
2017-01-24 22:12:39,870 [myid:] - INFO [Thread-12:FourLetterWordMain@43] - connecting to 127.0.0.1 11233
2017-01-24 22:12:39,871 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45892
2017-01-24 22:12:39,871 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:45892
2017-01-24 22:12:39,871 [myid:2] - INFO [Thread-93:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45892 (no session established for client)
2017-01-24 22:12:39,948 [myid:] - WARN [Thread-12-SendThread(localhost:11227):ClientCnxn$SendThread@957] - SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: '/data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test2999887027410032136.junit.dir/jaas.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.
2017-01-24 22:12:39,949 [myid:] - INFO [Thread-12-SendThread(localhost:11227):ClientCnxn$SendThread@975] - Opening socket connection to server localhost/127.0.0.1:11227
2017-01-24 22:12:39,949 [myid:0] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11227:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:42923
2017-01-24 22:12:39,949 [myid:] - INFO [Thread-12-SendThread(localhost:11227):ClientCnxn$SendThread@852] - Socket connection established, initiating session, client: /127.0.0.1:42923, server: localhost/127.0.0.1:11227
2017-01-24 22:12:39,949 [myid:0] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11227:NIOServerCnxn@354] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2017-01-24 22:12:39,950 [myid:0] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11227:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:42923 (no session established for client)
2017-01-24 22:12:39,950 [myid:] - INFO [Thread-12-SendThread(localhost:11227):ClientCnxn$SendThread@1098] - Unable to read additional data from server sessionid 0x159d440f0ed0000, likely server has closed socket, closing socket connection and attempting reconnect
2017-01-24 22:12:40,122 [myid:] - INFO [Thread-12:FourLetterWordMain@43] - connecting to 127.0.0.1 11233
2017-01-24 22:12:40,122 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45894
2017-01-24 22:12:40,123 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:45894
2017-01-24 22:12:40,123 [myid:2] - INFO [Thread-94:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45894 (no session established for client)
2017-01-24 22:12:40,374 [myid:] - INFO [Thread-12:FourLetterWordMain@43] - connecting to 127.0.0.1 11233
2017-01-24 22:12:40,375 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45895
2017-01-24 22:12:40,375 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:45895
2017-01-24 22:12:40,375 [myid:2] - INFO [Thread-95:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45895 (no session established for client)
2017-01-24 22:12:40,626 [myid:] - INFO [Thread-12:FourLetterWordMain@43] - connecting to 127.0.0.1 11233
2017-01-24 22:12:40,626 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45896
2017-01-24 22:12:40,627 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:45896
2017-01-24 22:12:40,627 [myid:2] - INFO [Thread-96:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45896 (no session established for client)
2017-01-24 22:12:40,877 [myid:] - INFO [Thread-12:FourLetterWordMain@43] - connecting to 127.0.0.1 11233
2017-01-24 22:12:40,878 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45897
2017-01-24 22:12:40,878 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:45897
2017-01-24 22:12:40,879 [myid:2] - INFO [Thread-97:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45897 (no session established for client)
2017-01-24 22:12:41,129 [myid:] - INFO [Thread-12:FourLetterWordMain@43] - connecting to 127.0.0.1 11233
2017-01-24 22:12:41,130 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45898
2017-01-24 22:12:41,130 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:45898
2017-01-24 22:12:41,131 [myid:2] - INFO [Thread-98:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45898 (no session established for client)
2017-01-24 22:12:41,205 [myid:] - WARN [Thread-12-SendThread(localhost:11230):ClientCnxn$SendThread@957] - SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: '/data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test2999887027410032136.junit.dir/jaas.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.
2017-01-24 22:12:41,206 [myid:] - INFO [Thread-12-SendThread(localhost:11230):ClientCnxn$SendThread@975] - Opening socket connection to server localhost/127.0.0.1:11230
2017-01-24 22:12:41,206 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11230:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:52430
2017-01-24 22:12:41,206 [myid:] - INFO [Thread-12-SendThread(localhost:11230):ClientCnxn$SendThread@852] - Socket connection established, initiating session, client: /127.0.0.1:52430, server: localhost/127.0.0.1:11230
2017-01-24 22:12:41,207 [myid:1] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11230:NIOServerCnxn@354] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2017-01-24 22:12:41,207 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11230:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:52430 (no session established for client)
2017-01-24 22:12:41,207 [myid:] - INFO [Thread-12-SendThread(localhost:11230):ClientCnxn$SendThread@1098] - Unable to read additional data from server sessionid 0x159d440f0ed0000, likely server has closed socket, closing socket connection and attempting reconnect
2017-01-24 22:12:41,257 [myid:1] - WARN [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11224:QuorumCnxManager@559] - Cannot open channel to 0 at election address localhost/127.0.0.1:11223
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:538)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:579)
at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:769)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:827)
2017-01-24 22:12:41,258 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11224:FastLeaderElection@778] - Notification time out: 12800
2017-01-24 22:12:41,381 [myid:] - INFO [Thread-12:FourLetterWordMain@43] - connecting to 127.0.0.1 11233
2017-01-24 22:12:41,381 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45901
2017-01-24 22:12:41,382 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:45901
2017-01-24 22:12:41,382 [myid:2] - INFO [Thread-99:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45901 (no session established for client)
2017-01-24 22:12:41,633 [myid:] - INFO [Thread-12:FourLetterWordMain@43] - connecting to 127.0.0.1 11233
2017-01-24 22:12:41,633 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45902
2017-01-24 22:12:41,633 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:45902
2017-01-24 22:12:41,634 [myid:2] - INFO [Thread-100:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45902 (no session established for client)
2017-01-24 22:12:41,741 [myid:] - WARN [Thread-12-SendThread(localhost:11233):ClientCnxn$SendThread@957] - SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: '/data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test2999887027410032136.junit.dir/jaas.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.
2017-01-24 22:12:41,742 [myid:] - INFO [Thread-12-SendThread(localhost:11233):ClientCnxn$SendThread@975] - Opening socket connection to server localhost/127.0.0.1:11233
2017-01-24 22:12:41,742 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45903
2017-01-24 22:12:41,742 [myid:] - INFO [Thread-12-SendThread(localhost:11233):ClientCnxn$SendThread@852] - Socket connection established, initiating session, client: /127.0.0.1:45903, server: localhost/127.0.0.1:11233
2017-01-24 22:12:41,742 [myid:2] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@354] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2017-01-24 22:12:41,742 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45903 (no session established for client)
2017-01-24 22:12:41,743 [myid:] - INFO [Thread-12-SendThread(localhost:11233):ClientCnxn$SendThread@1098] - Unable to read additional data from server sessionid 0x159d440f0ed0000, likely server has closed socket, closing socket connection and attempting reconnect
2017-01-24 22:12:41,884 [myid:] - INFO [Thread-12:FourLetterWordMain@43] - connecting to 127.0.0.1 11233
2017-01-24 22:12:41,885 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45904
2017-01-24 22:12:41,885 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:45904
2017-01-24 22:12:41,886 [myid:2] - INFO [Thread-101:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45904 (no session established for client)
2017-01-24 22:12:42,136 [myid:] - INFO [Thread-12:FourLetterWordMain@43] - connecting to 127.0.0.1 11233
2017-01-24 22:12:42,137 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45905
2017-01-24 22:12:42,137 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:45905
2017-01-24 22:12:42,137 [myid:2] - INFO [Thread-102:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45905 (no session established for client)
2017-01-24 22:12:42,388 [myid:] - INFO [Thread-12:FourLetterWordMain@43] - connecting to 127.0.0.1 11233
2017-01-24 22:12:42,388 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45906
2017-01-24 22:12:42,389 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:45906
2017-01-24 22:12:42,389 [myid:2] - INFO [Thread-103:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45906 (no session established for client)
2017-01-24 22:12:42,564 [myid:] - WARN [Thread-12-SendThread(localhost:11227):ClientCnxn$SendThread@957] - SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: '/data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test2999887027410032136.junit.dir/jaas.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.
2017-01-24 22:12:42,565 [myid:] - INFO [Thread-12-SendThread(localhost:11227):ClientCnxn$SendThread@975] - Opening socket connection to server localhost/127.0.0.1:11227
2017-01-24 22:12:42,565 [myid:0] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11227:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:42937
2017-01-24 22:12:42,565 [myid:] - INFO [Thread-12-SendThread(localhost:11227):ClientCnxn$SendThread@852] - Socket connection established, initiating session, client: /127.0.0.1:42937, server: localhost/127.0.0.1:11227
2017-01-24 22:12:42,565 [myid:0] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11227:NIOServerCnxn@354] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2017-01-24 22:12:42,566 [myid:0] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11227:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:42937 (no session established for client)
2017-01-24 22:12:42,566 [myid:] - INFO [Thread-12-SendThread(localhost:11227):ClientCnxn$SendThread@1098] - Unable to read additional data from server sessionid 0x159d440f0ed0000, likely server has closed socket, closing socket connection and attempting reconnect
2017-01-24 22:12:42,639 [myid:] - INFO [Thread-12:FourLetterWordMain@43] - connecting to 127.0.0.1 11233
2017-01-24 22:12:42,640 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45908
2017-01-24 22:12:42,640 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:45908
2017-01-24 22:12:42,641 [myid:2] - INFO [Thread-104:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45908 (no session established for client)
2017-01-24 22:12:42,891 [myid:] - INFO [Thread-12:FourLetterWordMain@43] - connecting to 127.0.0.1 11233
2017-01-24 22:12:42,892 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45909
2017-01-24 22:12:42,892 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:45909
2017-01-24 22:12:42,893 [myid:2] - INFO [Thread-105:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45909 (no session established for client)
2017-01-24 22:12:43,143 [myid:] - INFO [Thread-12:FourLetterWordMain@43] - connecting to 127.0.0.1 11233
2017-01-24 22:12:43,143 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45910
2017-01-24 22:12:43,144 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:45910
2017-01-24 22:12:43,145 [myid:2] - INFO [Thread-106:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45910 (no session established for client)
2017-01-24 22:12:43,395 [myid:] - INFO [Thread-12:FourLetterWordMain@43] - connecting to 127.0.0.1 11233
2017-01-24 22:12:43,396 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45911
2017-01-24 22:12:43,396 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:45911
2017-01-24 22:12:43,396 [myid:2] - INFO [Thread-107:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45911 (no session established for client)
2017-01-24 22:12:43,647 [myid:] - INFO [Thread-12:FourLetterWordMain@43] - connecting to 127.0.0.1 11233
2017-01-24 22:12:43,647 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45912
2017-01-24 22:12:43,647 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:45912
2017-01-24 22:12:43,648 [myid:2] - INFO [Thread-108:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45912 (no session established for client)
2017-01-24 22:12:43,667 [myid:] - WARN [Thread-12-SendThread(localhost:11230):ClientCnxn$SendThread@957] - SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: '/data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test2999887027410032136.junit.dir/jaas.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.
2017-01-24 22:12:43,667 [myid:] - INFO [Thread-12-SendThread(localhost:11230):ClientCnxn$SendThread@975] - Opening socket connection to server localhost/127.0.0.1:11230
2017-01-24 22:12:43,668 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11230:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:52444
2017-01-24 22:12:43,668 [myid:] - INFO [Thread-12-SendThread(localhost:11230):ClientCnxn$SendThread@852] - Socket connection established, initiating session, client: /127.0.0.1:52444, server: localhost/127.0.0.1:11230
2017-01-24 22:12:43,668 [myid:1] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11230:NIOServerCnxn@354] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2017-01-24 22:12:43,668 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11230:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:52444 (no session established for client)
2017-01-24 22:12:43,668 [myid:] - INFO [Thread-12-SendThread(localhost:11230):ClientCnxn$SendThread@1098] - Unable to read additional data from server sessionid 0x159d440f0ed0000, likely server has closed socket, closing socket connection and attempting reconnect
2017-01-24 22:12:43,898 [myid:] - INFO [Thread-12:FourLetterWordMain@43] - connecting to 127.0.0.1 11233
2017-01-24 22:12:43,899 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45914
2017-01-24 22:12:43,899 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:45914
2017-01-24 22:12:43,900 [myid:2] - INFO [Thread-109:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45914 (no session established for client)
2017-01-24 22:12:44,150 [myid:] - INFO [Thread-12:FourLetterWordMain@43] - connecting to 127.0.0.1 11233
2017-01-24 22:12:44,151 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45915
2017-01-24 22:12:44,151 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:45915
2017-01-24 22:12:44,151 [myid:2] - INFO [Thread-110:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45915 (no session established for client)
2017-01-24 22:12:44,402 [myid:] - INFO [Thread-12:FourLetterWordMain@43] - connecting to 127.0.0.1 11233
2017-01-24 22:12:44,403 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45916
2017-01-24 22:12:44,403 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:45916
2017-01-24 22:12:44,404 [myid:2] - INFO [Thread-111:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45916 (no session established for client)
2017-01-24 22:12:44,525 [myid:] - WARN [Thread-12-SendThread(localhost:11233):ClientCnxn$SendThread@957] - SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: '/data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test2999887027410032136.junit.dir/jaas.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.
2017-01-24 22:12:44,525 [myid:] - INFO [Thread-12-SendThread(localhost:11233):ClientCnxn$SendThread@975] - Opening socket connection to server localhost/127.0.0.1:11233
2017-01-24 22:12:44,526 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45917
2017-01-24 22:12:44,526 [myid:] - INFO [Thread-12-SendThread(localhost:11233):ClientCnxn$SendThread@852] - Socket connection established, initiating session, client: /127.0.0.1:45917, server: localhost/127.0.0.1:11233
2017-01-24 22:12:44,526 [myid:2] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@354] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2017-01-24 22:12:44,526 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45917 (no session established for client)
2017-01-24 22:12:44,526 [myid:] - INFO [Thread-12-SendThread(localhost:11233):ClientCnxn$SendThread@1098] - Unable to read additional data from server sessionid 0x159d440f0ed0000, likely server has closed socket, closing socket connection and attempting reconnect
2017-01-24 22:12:44,654 [myid:] - INFO [Thread-12:FourLetterWordMain@43] - connecting to 127.0.0.1 11233
2017-01-24 22:12:44,655 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45918
2017-01-24 22:12:44,655 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:45918
2017-01-24 22:12:44,655 [myid:2] - INFO [Thread-112:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45918 (no session established for client)
2017-01-24 22:12:44,906 [myid:] - INFO [Thread-12:FourLetterWordMain@43] - connecting to 127.0.0.1 11233
2017-01-24 22:12:44,906 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45919
2017-01-24 22:12:44,907 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:45919
2017-01-24 22:12:44,907 [myid:2] - INFO [Thread-113:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45919 (no session established for client)
2017-01-24 22:12:45,158 [myid:] - INFO [Thread-12:FourLetterWordMain@43] - connecting to 127.0.0.1 11233
2017-01-24 22:12:45,158 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45920
2017-01-24 22:12:45,158 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:45920
2017-01-24 22:12:45,159 [myid:2] - INFO [Thread-114:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45920 (no session established for client)
2017-01-24 22:12:45,409 [myid:] - INFO [Thread-12:FourLetterWordMain@43] - connecting to 127.0.0.1 11233
2017-01-24 22:12:45,410 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45921
2017-01-24 22:12:45,410 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:45921
2017-01-24 22:12:45,411 [myid:2] - INFO [Thread-115:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45921 (no session established for client)
2017-01-24 22:12:45,556 [myid:] - WARN [Thread-12-SendThread(localhost:11227):ClientCnxn$SendThread@957] - SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: '/data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test2999887027410032136.junit.dir/jaas.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.
2017-01-24 22:12:45,556 [myid:] - INFO [Thread-12-SendThread(localhost:11227):ClientCnxn$SendThread@975] - Opening socket connection to server localhost/127.0.0.1:11227
2017-01-24 22:12:45,557 [myid:0] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11227:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:42952
2017-01-24 22:12:45,557 [myid:] - INFO [Thread-12-SendThread(localhost:11227):ClientCnxn$SendThread@852] - Socket connection established, initiating session, client: /127.0.0.1:42952, server: localhost/127.0.0.1:11227
2017-01-24 22:12:45,557 [myid:0] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11227:NIOServerCnxn@354] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2017-01-24 22:12:45,557 [myid:0] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11227:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:42952 (no session established for client)
2017-01-24 22:12:45,560 [myid:] - INFO [Thread-12-SendThread(localhost:11227):ClientCnxn$SendThread@1098] - Unable to read additional data from server sessionid 0x159d440f0ed0000, likely server has closed socket, closing socket connection and attempting reconnect
2017-01-24 22:12:45,661 [myid:] - INFO [Thread-12:FourLetterWordMain@43] - connecting to 127.0.0.1 11233
2017-01-24 22:12:45,661 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45923
2017-01-24 22:12:45,662 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:45923
2017-01-24 22:12:45,662 [myid:2] - INFO [Thread-116:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45923 (no session established for client)
2017-01-24 22:12:45,913 [myid:] - INFO [Thread-12:FourLetterWordMain@43] - connecting to 127.0.0.1 11233
2017-01-24 22:12:45,913 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45924
2017-01-24 22:12:45,913 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:45924
2017-01-24 22:12:45,914 [myid:2] - INFO [Thread-117:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45924 (no session established for client)
2017-01-24 22:12:46,164 [myid:] - INFO [Thread-12:FourLetterWordMain@43] - connecting to 127.0.0.1 11233
2017-01-24 22:12:46,165 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45925
2017-01-24 22:12:46,165 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:45925
2017-01-24 22:12:46,166 [myid:2] - INFO [Thread-118:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45925 (no session established for client)
2017-01-24 22:12:46,416 [myid:] - INFO [Thread-12:FourLetterWordMain@43] - connecting to 127.0.0.1 11233
2017-01-24 22:12:46,417 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45926
2017-01-24 22:12:46,417 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:45926
2017-01-24 22:12:46,417 [myid:2] - INFO [Thread-119:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45926 (no session established for client)
2017-01-24 22:12:46,668 [myid:] - INFO [Thread-12:FourLetterWordMain@43] - connecting to 127.0.0.1 11233
2017-01-24 22:12:46,668 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45927
2017-01-24 22:12:46,669 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:45927
2017-01-24 22:12:46,669 [myid:2] - INFO [Thread-120:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45927 (no session established for client)
2017-01-24 22:12:46,919 [myid:] - INFO [Thread-12:FourLetterWordMain@43] - connecting to 127.0.0.1 11233
2017-01-24 22:12:46,920 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45928
2017-01-24 22:12:46,920 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:45928
2017-01-24 22:12:46,921 [myid:2] - INFO [Thread-121:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45928 (no session established for client)
2017-01-24 22:12:46,935 [myid:] - WARN [Thread-12-SendThread(localhost:11230):ClientCnxn$SendThread@957] - SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: '/data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test2999887027410032136.junit.dir/jaas.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.
2017-01-24 22:12:46,935 [myid:] - INFO [Thread-12-SendThread(localhost:11230):ClientCnxn$SendThread@975] - Opening socket connection to server localhost/127.0.0.1:11230
2017-01-24 22:12:46,935 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11230:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:52460
2017-01-24 22:12:46,935 [myid:] - INFO [Thread-12-SendThread(localhost:11230):ClientCnxn$SendThread@852] - Socket connection established, initiating session, client: /127.0.0.1:52460, server: localhost/127.0.0.1:11230
2017-01-24 22:12:46,936 [myid:1] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11230:NIOServerCnxn@354] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2017-01-24 22:12:46,936 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11230:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:52460 (no session established for client)
2017-01-24 22:12:46,936 [myid:] - INFO [Thread-12-SendThread(localhost:11230):ClientCnxn$SendThread@1098] - Unable to read additional data from server sessionid 0x159d440f0ed0000, likely server has closed socket, closing socket connection and attempting reconnect
2017-01-24 22:12:47,171 [myid:] - INFO [Thread-12:FourLetterWordMain@43] - connecting to 127.0.0.1 11233
2017-01-24 22:12:47,172 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45930
2017-01-24 22:12:47,172 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:45930
2017-01-24 22:12:47,172 [myid:2] - INFO [Thread-122:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45930 (no session established for client)
2017-01-24 22:12:47,423 [myid:] - INFO [Thread-12:FourLetterWordMain@43] - connecting to 127.0.0.1 11233
2017-01-24 22:12:47,423 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45931
2017-01-24 22:12:47,424 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:45931
2017-01-24 22:12:47,424 [myid:2] - INFO [Thread-123:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45931 (no session established for client)
2017-01-24 22:12:47,470 [myid:] - WARN [Thread-12-SendThread(localhost:11233):ClientCnxn$SendThread@957] - SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: '/data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test2999887027410032136.junit.dir/jaas.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.
2017-01-24 22:12:47,470 [myid:] - INFO [Thread-12-SendThread(localhost:11233):ClientCnxn$SendThread@975] - Opening socket connection to server localhost/127.0.0.1:11233
2017-01-24 22:12:47,470 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45932
2017-01-24 22:12:47,470 [myid:] - INFO [Thread-12-SendThread(localhost:11233):ClientCnxn$SendThread@852] - Socket connection established, initiating session, client: /127.0.0.1:45932, server: localhost/127.0.0.1:11233
2017-01-24 22:12:47,477 [myid:2] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@354] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2017-01-24 22:12:47,477 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45932 (no session established for client)
2017-01-24 22:12:47,477 [myid:] - INFO [Thread-12-SendThread(localhost:11233):ClientCnxn$SendThread@1098] - Unable to read additional data from server sessionid 0x159d440f0ed0000, likely server has closed socket, closing socket connection and attempting reconnect
2017-01-24 22:12:47,675 [myid:] - INFO [Thread-12:FourLetterWordMain@43] - connecting to 127.0.0.1 11233
2017-01-24 22:12:47,675 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45933
2017-01-24 22:12:47,675 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:45933
2017-01-24 22:12:47,676 [myid:2] - INFO [Thread-124:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45933 (no session established for client)
2017-01-24 22:12:47,926 [myid:] - INFO [Thread-12:FourLetterWordMain@43] - connecting to 127.0.0.1 11233
2017-01-24 22:12:47,927 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45934
2017-01-24 22:12:47,927 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:45934
2017-01-24 22:12:47,928 [myid:2] - INFO [Thread-125:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45934 (no session established for client)
2017-01-24 22:12:48,178 [myid:] - INFO [Thread-12:FourLetterWordMain@43] - connecting to 127.0.0.1 11233
2017-01-24 22:12:48,179 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45935
2017-01-24 22:12:48,179 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:45935
2017-01-24 22:12:48,179 [myid:2] - INFO [Thread-126:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45935 (no session established for client)
2017-01-24 22:12:48,430 [myid:] - INFO [Thread-12:FourLetterWordMain@43] - connecting to 127.0.0.1 11233
2017-01-24 22:12:48,430 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45936
2017-01-24 22:12:48,430 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:45936
2017-01-24 22:12:48,431 [myid:2] - INFO [Thread-127:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45936 (no session established for client)
2017-01-24 22:12:48,469 [myid:] - WARN [Thread-12-SendThread(localhost:11227):ClientCnxn$SendThread@957] - SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: '/data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test2999887027410032136.junit.dir/jaas.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.
2017-01-24 22:12:48,469 [myid:] - INFO [Thread-12-SendThread(localhost:11227):ClientCnxn$SendThread@975] - Opening socket connection to server localhost/127.0.0.1:11227
2017-01-24 22:12:48,469 [myid:0] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11227:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:42967
2017-01-24 22:12:48,469 [myid:] - INFO [Thread-12-SendThread(localhost:11227):ClientCnxn$SendThread@852] - Socket connection established, initiating session, client: /127.0.0.1:42967, server: localhost/127.0.0.1:11227
2017-01-24 22:12:48,470 [myid:0] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11227:NIOServerCnxn@354] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2017-01-24 22:12:48,470 [myid:0] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11227:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:42967 (no session established for client)
2017-01-24 22:12:48,470 [myid:] - INFO [Thread-12-SendThread(localhost:11227):ClientCnxn$SendThread@1098] - Unable to read additional data from server sessionid 0x159d440f0ed0000, likely server has closed socket, closing socket connection and attempting reconnect
2017-01-24 22:12:48,681 [myid:] - INFO [Thread-12:FourLetterWordMain@43] - connecting to 127.0.0.1 11233
2017-01-24 22:12:48,682 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45938
2017-01-24 22:12:48,682 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:45938
2017-01-24 22:12:48,683 [myid:2] - INFO [Thread-128:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45938 (no session established for client)
2017-01-24 22:12:48,933 [myid:] - INFO [Thread-12:FourLetterWordMain@43] - connecting to 127.0.0.1 11233
2017-01-24 22:12:48,934 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45939
2017-01-24 22:12:48,934 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:45939
2017-01-24 22:12:48,935 [myid:2] - INFO [Thread-129:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45939 (no session established for client)
2017-01-24 22:12:49,185 [myid:] - INFO [Thread-12:FourLetterWordMain@43] - connecting to 127.0.0.1 11233
2017-01-24 22:12:49,186 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45940
2017-01-24 22:12:49,186 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:45940
2017-01-24 22:12:49,186 [myid:2] - INFO [Thread-130:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45940 (no session established for client)
2017-01-24 22:12:49,437 [myid:] - INFO [Thread-12:FourLetterWordMain@43] - connecting to 127.0.0.1 11233
2017-01-24 22:12:49,437 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45941
2017-01-24 22:12:49,437 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:45941
2017-01-24 22:12:49,438 [myid:2] - INFO [Thread-131:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45941 (no session established for client)
2017-01-24 22:12:49,688 [myid:] - INFO [Thread-12:FourLetterWordMain@43] - connecting to 127.0.0.1 11233
2017-01-24 22:12:49,689 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45942
2017-01-24 22:12:49,689 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:45942
2017-01-24 22:12:49,690 [myid:2] - INFO [Thread-132:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45942 (no session established for client)
2017-01-24 22:12:49,940 [myid:] - INFO [Thread-12:FourLetterWordMain@43] - connecting to 127.0.0.1 11233
2017-01-24 22:12:49,940 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45943
2017-01-24 22:12:49,941 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:45943
2017-01-24 22:12:49,941 [myid:2] - INFO [Thread-133:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45943 (no session established for client)
2017-01-24 22:12:50,192 [myid:] - INFO [Thread-12:FourLetterWordMain@43] - connecting to 127.0.0.1 11233
2017-01-24 22:12:50,192 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45944
2017-01-24 22:12:50,192 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:45944
2017-01-24 22:12:50,193 [myid:2] - INFO [Thread-134:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45944 (no session established for client)
2017-01-24 22:12:50,213 [myid:] - WARN [Thread-12-SendThread(localhost:11230):ClientCnxn$SendThread@957] - SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: '/data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test2999887027410032136.junit.dir/jaas.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.
2017-01-24 22:12:50,213 [myid:] - INFO [Thread-12-SendThread(localhost:11230):ClientCnxn$SendThread@975] - Opening socket connection to server localhost/127.0.0.1:11230
2017-01-24 22:12:50,213 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11230:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:52476
2017-01-24 22:12:50,213 [myid:] - INFO [Thread-12-SendThread(localhost:11230):ClientCnxn$SendThread@852] - Socket connection established, initiating session, client: /127.0.0.1:52476, server: localhost/127.0.0.1:11230
2017-01-24 22:12:50,214 [myid:1] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11230:NIOServerCnxn@354] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2017-01-24 22:12:50,214 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11230:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:52476 (no session established for client)
2017-01-24 22:12:50,214 [myid:] - INFO [Thread-12-SendThread(localhost:11230):ClientCnxn$SendThread@1098] - Unable to read additional data from server sessionid 0x159d440f0ed0000, likely server has closed socket, closing socket connection and attempting reconnect
2017-01-24 22:12:50,314 [myid:] - WARN [Thread-12-SendThread(localhost:11233):ClientCnxn$SendThread@957] - SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: '/data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test2999887027410032136.junit.dir/jaas.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.
2017-01-24 22:12:50,314 [myid:] - INFO [Thread-12-SendThread(localhost:11233):ClientCnxn$SendThread@975] - Opening socket connection to server localhost/127.0.0.1:11233
2017-01-24 22:12:50,315 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45946
2017-01-24 22:12:50,315 [myid:] - INFO [Thread-12-SendThread(localhost:11233):ClientCnxn$SendThread@852] - Socket connection established, initiating session, client: /127.0.0.1:45946, server: localhost/127.0.0.1:11233
2017-01-24 22:12:50,315 [myid:2] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@354] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2017-01-24 22:12:50,315 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45946 (no session established for client)
2017-01-24 22:12:50,316 [myid:] - INFO [Thread-12-SendThread(localhost:11233):ClientCnxn$SendThread@1098] - Unable to read additional data from server sessionid 0x159d440f0ed0000, likely server has closed socket, closing socket connection and attempting reconnect
2017-01-24 22:12:50,443 [myid:] - INFO [Thread-12:FourLetterWordMain@43] - connecting to 127.0.0.1 11233
2017-01-24 22:12:50,444 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45947
2017-01-24 22:12:50,444 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:45947
2017-01-24 22:12:50,445 [myid:2] - INFO [Thread-135:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45947 (no session established for client)
2017-01-24 22:12:50,695 [myid:] - INFO [Thread-12:FourLetterWordMain@43] - connecting to 127.0.0.1 11233
2017-01-24 22:12:50,696 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45948
2017-01-24 22:12:50,696 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:45948
2017-01-24 22:12:50,696 [myid:2] - INFO [Thread-136:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45948 (no session established for client)
2017-01-24 22:12:50,947 [myid:] - INFO [Thread-12:FourLetterWordMain@43] - connecting to 127.0.0.1 11233
2017-01-24 22:12:50,947 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45949
2017-01-24 22:12:50,947 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:45949
2017-01-24 22:12:50,948 [myid:2] - INFO [Thread-137:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45949 (no session established for client)
2017-01-24 22:12:50,968 [myid:] - WARN [Thread-12-SendThread(localhost:11227):ClientCnxn$SendThread@957] - SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: '/data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test2999887027410032136.junit.dir/jaas.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.
2017-01-24 22:12:50,968 [myid:] - INFO [Thread-12-SendThread(localhost:11227):ClientCnxn$SendThread@975] - Opening socket connection to server localhost/127.0.0.1:11227
2017-01-24 22:12:50,969 [myid:0] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11227:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:42980
2017-01-24 22:12:50,969 [myid:] - INFO [Thread-12-SendThread(localhost:11227):ClientCnxn$SendThread@852] - Socket connection established, initiating session, client: /127.0.0.1:42980, server: localhost/127.0.0.1:11227
2017-01-24 22:12:50,969 [myid:0] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11227:NIOServerCnxn@354] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2017-01-24 22:12:50,969 [myid:0] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11227:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:42980 (no session established for client)
2017-01-24 22:12:50,970 [myid:] - INFO [Thread-12-SendThread(localhost:11227):ClientCnxn$SendThread@1098] - Unable to read additional data from server sessionid 0x159d440f0ed0000, likely server has closed socket, closing socket connection and attempting reconnect
2017-01-24 22:12:51,198 [myid:] - INFO [Thread-12:FourLetterWordMain@43] - connecting to 127.0.0.1 11233
2017-01-24 22:12:51,199 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45951
2017-01-24 22:12:51,199 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:45951
2017-01-24 22:12:51,200 [myid:2] - INFO [Thread-138:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45951 (no session established for client)
2017-01-24 22:12:51,450 [myid:] - INFO [Thread-12:FourLetterWordMain@43] - connecting to 127.0.0.1 11233
2017-01-24 22:12:51,451 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45952
2017-01-24 22:12:51,451 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:45952
2017-01-24 22:12:51,452 [myid:2] - INFO [Thread-139:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45952 (no session established for client)
2017-01-24 22:12:51,702 [myid:] - INFO [Thread-12:FourLetterWordMain@43] - connecting to 127.0.0.1 11233
2017-01-24 22:12:51,703 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45953
2017-01-24 22:12:51,703 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:45953
2017-01-24 22:12:51,704 [myid:2] - INFO [Thread-140:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45953 (no session established for client)
2017-01-24 22:12:51,954 [myid:] - INFO [Thread-12:FourLetterWordMain@43] - connecting to 127.0.0.1 11233
2017-01-24 22:12:51,955 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45954
2017-01-24 22:12:51,955 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:45954
2017-01-24 22:12:51,955 [myid:2] - INFO [Thread-141:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45954 (no session established for client)
2017-01-24 22:12:52,206 [myid:] - INFO [Thread-12:FourLetterWordMain@43] - connecting to 127.0.0.1 11233
2017-01-24 22:12:52,206 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45955
2017-01-24 22:12:52,207 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:45955
2017-01-24 22:12:52,207 [myid:2] - INFO [Thread-142:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45955 (no session established for client)
2017-01-24 22:12:52,457 [myid:] - INFO [Thread-12:FourLetterWordMain@43] - connecting to 127.0.0.1 11233
2017-01-24 22:12:52,458 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45956
2017-01-24 22:12:52,459 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:45956
2017-01-24 22:12:52,459 [myid:2] - INFO [Thread-143:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45956 (no session established for client)
2017-01-24 22:12:52,477 [myid:] - WARN [Thread-12-SendThread(localhost:11230):ClientCnxn$SendThread@957] - SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: '/data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test2999887027410032136.junit.dir/jaas.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.
2017-01-24 22:12:52,478 [myid:] - INFO [Thread-12-SendThread(localhost:11230):ClientCnxn$SendThread@975] - Opening socket connection to server localhost/127.0.0.1:11230
2017-01-24 22:12:52,478 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11230:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:52488
2017-01-24 22:12:52,478 [myid:] - INFO [Thread-12-SendThread(localhost:11230):ClientCnxn$SendThread@852] - Socket connection established, initiating session, client: /127.0.0.1:52488, server: localhost/127.0.0.1:11230
2017-01-24 22:12:52,478 [myid:1] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11230:NIOServerCnxn@354] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2017-01-24 22:12:52,478 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11230:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:52488 (no session established for client)
2017-01-24 22:12:52,479 [myid:] - INFO [Thread-12-SendThread(localhost:11230):ClientCnxn$SendThread@1098] - Unable to read additional data from server sessionid 0x159d440f0ed0000, likely server has closed socket, closing socket connection and attempting reconnect
2017-01-24 22:12:52,710 [myid:] - INFO [Thread-12:FourLetterWordMain@43] - connecting to 127.0.0.1 11233
2017-01-24 22:12:52,710 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45958
2017-01-24 22:12:52,711 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:45958
2017-01-24 22:12:52,711 [myid:2] - INFO [Thread-144:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45958 (no session established for client)
2017-01-24 22:12:52,948 [myid:] - WARN [Thread-12-SendThread(localhost:11233):ClientCnxn$SendThread@957] - SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: '/data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test2999887027410032136.junit.dir/jaas.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.
2017-01-24 22:12:52,948 [myid:] - INFO [Thread-12-SendThread(localhost:11233):ClientCnxn$SendThread@975] - Opening socket connection to server localhost/127.0.0.1:11233
2017-01-24 22:12:52,949 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45959
2017-01-24 22:12:52,949 [myid:] - INFO [Thread-12-SendThread(localhost:11233):ClientCnxn$SendThread@852] - Socket connection established, initiating session, client: /127.0.0.1:45959, server: localhost/127.0.0.1:11233
2017-01-24 22:12:52,949 [myid:2] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@354] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2017-01-24 22:12:52,949 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45959 (no session established for client)
2017-01-24 22:12:52,949 [myid:] - INFO [Thread-12-SendThread(localhost:11233):ClientCnxn$SendThread@1098] - Unable to read additional data from server sessionid 0x159d440f0ed0000, likely server has closed socket, closing socket connection and attempting reconnect
2017-01-24 22:12:52,961 [myid:] - INFO [Thread-12:FourLetterWordMain@43] - connecting to 127.0.0.1 11233
2017-01-24 22:12:52,962 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45960
2017-01-24 22:12:52,962 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:45960
2017-01-24 22:12:52,962 [myid:2] - INFO [Thread-145:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45960 (no session established for client)
2017-01-24 22:12:53,213 [myid:] - INFO [Thread-12:FourLetterWordMain@43] - connecting to 127.0.0.1 11233
2017-01-24 22:12:53,213 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45961
2017-01-24 22:12:53,214 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:45961
2017-01-24 22:12:53,214 [myid:2] - INFO [Thread-146:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45961 (no session established for client)
2017-01-24 22:12:53,464 [myid:] - INFO [Thread-12:FourLetterWordMain@43] - connecting to 127.0.0.1 11233
2017-01-24 22:12:53,465 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45962
2017-01-24 22:12:53,465 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:45962
2017-01-24 22:12:53,466 [myid:2] - INFO [Thread-147:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45962 (no session established for client)
2017-01-24 22:12:53,716 [myid:] - INFO [Thread-12:FourLetterWordMain@43] - connecting to 127.0.0.1 11233
2017-01-24 22:12:53,717 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45963
2017-01-24 22:12:53,717 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:45963
2017-01-24 22:12:53,718 [myid:2] - INFO [Thread-148:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45963 (no session established for client)
2017-01-24 22:12:53,878 [myid:] - WARN [Thread-12-SendThread(localhost:11227):ClientCnxn$SendThread@957] - SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: '/data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test2999887027410032136.junit.dir/jaas.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.
2017-01-24 22:12:53,878 [myid:] - INFO [Thread-12-SendThread(localhost:11227):ClientCnxn$SendThread@975] - Opening socket connection to server localhost/127.0.0.1:11227
2017-01-24 22:12:53,879 [myid:0] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11227:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:42994
2017-01-24 22:12:53,879 [myid:] - INFO [Thread-12-SendThread(localhost:11227):ClientCnxn$SendThread@852] - Socket connection established, initiating session, client: /127.0.0.1:42994, server: localhost/127.0.0.1:11227
2017-01-24 22:12:53,879 [myid:0] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11227:NIOServerCnxn@354] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2017-01-24 22:12:53,879 [myid:0] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11227:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:42994 (no session established for client)
2017-01-24 22:12:53,879 [myid:] - INFO [Thread-12-SendThread(localhost:11227):ClientCnxn$SendThread@1098] - Unable to read additional data from server sessionid 0x159d440f0ed0000, likely server has closed socket, closing socket connection and attempting reconnect
2017-01-24 22:12:53,968 [myid:] - INFO [Thread-12:FourLetterWordMain@43] - connecting to 127.0.0.1 11233
2017-01-24 22:12:53,968 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45965
2017-01-24 22:12:53,969 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:45965
2017-01-24 22:12:53,969 [myid:2] - INFO [Thread-149:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45965 (no session established for client)
2017-01-24 22:12:54,058 [myid:1] - WARN [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11224:QuorumCnxManager@559] - Cannot open channel to 0 at election address localhost/127.0.0.1:11223
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:538)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:579)
at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:769)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:827)
2017-01-24 22:12:54,059 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11224:FastLeaderElection@778] - Notification time out: 25600
2017-01-24 22:12:54,220 [myid:] - INFO [Thread-12:FourLetterWordMain@43] - connecting to 127.0.0.1 11233
2017-01-24 22:12:54,220 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45967
2017-01-24 22:12:54,220 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:45967
2017-01-24 22:12:54,221 [myid:2] - INFO [Thread-150:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45967 (no session established for client)
2017-01-24 22:12:54,471 [myid:] - INFO [Thread-12:FourLetterWordMain@43] - connecting to 127.0.0.1 11233
2017-01-24 22:12:54,472 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45968
2017-01-24 22:12:54,472 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:45968
2017-01-24 22:12:54,473 [myid:2] - INFO [Thread-151:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45968 (no session established for client)
2017-01-24 22:12:54,723 [myid:] - INFO [Thread-12:FourLetterWordMain@43] - connecting to 127.0.0.1 11233
2017-01-24 22:12:54,723 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45969
2017-01-24 22:12:54,724 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:45969
2017-01-24 22:12:54,724 [myid:2] - INFO [Thread-152:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45969 (no session established for client)
2017-01-24 22:12:54,975 [myid:] - INFO [Thread-12:FourLetterWordMain@43] - connecting to 127.0.0.1 11233
2017-01-24 22:12:54,975 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45970
2017-01-24 22:12:54,975 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:45970
2017-01-24 22:12:54,976 [myid:2] - INFO [Thread-153:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45970 (no session established for client)
2017-01-24 22:12:55,110 [myid:] - WARN [Thread-12-SendThread(localhost:11230):ClientCnxn$SendThread@957] - SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: '/data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test2999887027410032136.junit.dir/jaas.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.
2017-01-24 22:12:55,110 [myid:] - INFO [Thread-12-SendThread(localhost:11230):ClientCnxn$SendThread@975] - Opening socket connection to server localhost/127.0.0.1:11230
2017-01-24 22:12:55,111 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11230:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:52502
2017-01-24 22:12:55,111 [myid:] - INFO [Thread-12-SendThread(localhost:11230):ClientCnxn$SendThread@852] - Socket connection established, initiating session, client: /127.0.0.1:52502, server: localhost/127.0.0.1:11230
2017-01-24 22:12:55,111 [myid:1] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11230:NIOServerCnxn@354] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2017-01-24 22:12:55,111 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11230:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:52502 (no session established for client)
2017-01-24 22:12:55,111 [myid:] - INFO [Thread-12-SendThread(localhost:11230):ClientCnxn$SendThread@1098] - Unable to read additional data from server sessionid 0x159d440f0ed0000, likely server has closed socket, closing socket connection and attempting reconnect
2017-01-24 22:12:55,226 [myid:] - INFO [Thread-12:FourLetterWordMain@43] - connecting to 127.0.0.1 11233
2017-01-24 22:12:55,227 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45972
2017-01-24 22:12:55,227 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:45972
2017-01-24 22:12:55,229 [myid:2] - INFO [Thread-154:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45972 (no session established for client)
2017-01-24 22:12:55,479 [myid:] - INFO [Thread-12:FourLetterWordMain@43] - connecting to 127.0.0.1 11233
2017-01-24 22:12:55,480 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45973
2017-01-24 22:12:55,480 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:45973
2017-01-24 22:12:55,481 [myid:2] - INFO [Thread-155:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45973 (no session established for client)
2017-01-24 22:12:55,718 [myid:] - WARN [Thread-12-SendThread(localhost:11233):ClientCnxn$SendThread@957] - SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: '/data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test2999887027410032136.junit.dir/jaas.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.
2017-01-24 22:12:55,718 [myid:] - INFO [Thread-12-SendThread(localhost:11233):ClientCnxn$SendThread@975] - Opening socket connection to server localhost/127.0.0.1:11233
2017-01-24 22:12:55,718 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45974
2017-01-24 22:12:55,719 [myid:] - INFO [Thread-12-SendThread(localhost:11233):ClientCnxn$SendThread@852] - Socket connection established, initiating session, client: /127.0.0.1:45974, server: localhost/127.0.0.1:11233
2017-01-24 22:12:55,719 [myid:2] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@354] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2017-01-24 22:12:55,719 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45974 (no session established for client)
2017-01-24 22:12:55,719 [myid:] - INFO [Thread-12-SendThread(localhost:11233):ClientCnxn$SendThread@1098] - Unable to read additional data from server sessionid 0x159d440f0ed0000, likely server has closed socket, closing socket connection and attempting reconnect
2017-01-24 22:12:55,731 [myid:] - INFO [Thread-12:FourLetterWordMain@43] - connecting to 127.0.0.1 11233
2017-01-24 22:12:55,731 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45975
2017-01-24 22:12:55,731 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:45975
2017-01-24 22:12:55,732 [myid:2] - INFO [Thread-156:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45975 (no session established for client)
2017-01-24 22:12:55,982 [myid:] - INFO [Thread-12:FourLetterWordMain@43] - connecting to 127.0.0.1 11233
2017-01-24 22:12:55,983 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45976
2017-01-24 22:12:55,983 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:45976
2017-01-24 22:12:55,984 [myid:2] - INFO [Thread-157:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45976 (no session established for client)
2017-01-24 22:12:56,115 [myid:] - WARN [Thread-12-SendThread(localhost:11227):ClientCnxn$SendThread@957] - SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: '/data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test2999887027410032136.junit.dir/jaas.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.
2017-01-24 22:12:56,115 [myid:] - INFO [Thread-12-SendThread(localhost:11227):ClientCnxn$SendThread@975] - Opening socket connection to server localhost/127.0.0.1:11227
2017-01-24 22:12:56,115 [myid:0] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11227:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:43007
2017-01-24 22:12:56,115 [myid:] - INFO [Thread-12-SendThread(localhost:11227):ClientCnxn$SendThread@852] - Socket connection established, initiating session, client: /127.0.0.1:43007, server: localhost/127.0.0.1:11227
2017-01-24 22:12:56,116 [myid:0] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11227:NIOServerCnxn@354] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2017-01-24 22:12:56,116 [myid:0] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11227:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:43007 (no session established for client)
2017-01-24 22:12:56,116 [myid:] - INFO [Thread-12-SendThread(localhost:11227):ClientCnxn$SendThread@1098] - Unable to read additional data from server sessionid 0x159d440f0ed0000, likely server has closed socket, closing socket connection and attempting reconnect
2017-01-24 22:12:56,234 [myid:] - INFO [Thread-12:FourLetterWordMain@43] - connecting to 127.0.0.1 11233
2017-01-24 22:12:56,235 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45978
2017-01-24 22:12:56,235 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:45978
2017-01-24 22:12:56,236 [myid:2] - INFO [Thread-158:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45978 (no session established for client)
2017-01-24 22:12:56,486 [myid:] - INFO [Thread-12:FourLetterWordMain@43] - connecting to 127.0.0.1 11233
2017-01-24 22:12:56,487 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45979
2017-01-24 22:12:56,487 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:45979
2017-01-24 22:12:56,488 [myid:2] - INFO [Thread-159:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45979 (no session established for client)
2017-01-24 22:12:56,738 [myid:] - INFO [Thread-12:FourLetterWordMain@43] - connecting to 127.0.0.1 11233
2017-01-24 22:12:56,738 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45980
2017-01-24 22:12:56,739 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:45980
2017-01-24 22:12:56,739 [myid:2] - INFO [Thread-160:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45980 (no session established for client)
2017-01-24 22:12:56,989 [myid:] - INFO [Thread-12:FourLetterWordMain@43] - connecting to 127.0.0.1 11233
2017-01-24 22:12:56,990 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45981
2017-01-24 22:12:56,990 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:45981
2017-01-24 22:12:56,991 [myid:2] - INFO [Thread-161:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45981 (no session established for client)
2017-01-24 22:12:57,241 [myid:] - INFO [Thread-12:FourLetterWordMain@43] - connecting to 127.0.0.1 11233
2017-01-24 22:12:57,242 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45982
2017-01-24 22:12:57,242 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:45982
2017-01-24 22:12:57,242 [myid:2] - INFO [Thread-162:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45982 (no session established for client)
2017-01-24 22:12:57,493 [myid:] - INFO [Thread-12:FourLetterWordMain@43] - connecting to 127.0.0.1 11233
2017-01-24 22:12:57,493 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45983
2017-01-24 22:12:57,494 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:45983
2017-01-24 22:12:57,494 [myid:2] - INFO [Thread-163:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45983 (no session established for client)
2017-01-24 22:12:57,744 [myid:] - INFO [Thread-12:FourLetterWordMain@43] - connecting to 127.0.0.1 11233
2017-01-24 22:12:57,745 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45984
2017-01-24 22:12:57,745 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:45984
2017-01-24 22:12:57,746 [myid:2] - INFO [Thread-164:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45984 (no session established for client)
2017-01-24 22:12:57,885 [myid:] - WARN [Thread-12-SendThread(localhost:11230):ClientCnxn$SendThread@957] - SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: '/data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test2999887027410032136.junit.dir/jaas.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.
2017-01-24 22:12:57,885 [myid:] - INFO [Thread-12-SendThread(localhost:11230):ClientCnxn$SendThread@975] - Opening socket connection to server localhost/127.0.0.1:11230
2017-01-24 22:12:57,885 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11230:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:52516
2017-01-24 22:12:57,885 [myid:] - INFO [Thread-12-SendThread(localhost:11230):ClientCnxn$SendThread@852] - Socket connection established, initiating session, client: /127.0.0.1:52516, server: localhost/127.0.0.1:11230
2017-01-24 22:12:57,886 [myid:1] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11230:NIOServerCnxn@354] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2017-01-24 22:12:57,886 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11230:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:52516 (no session established for client)
2017-01-24 22:12:57,886 [myid:] - INFO [Thread-12-SendThread(localhost:11230):ClientCnxn$SendThread@1098] - Unable to read additional data from server sessionid 0x159d440f0ed0000, likely server has closed socket, closing socket connection and attempting reconnect
2017-01-24 22:12:57,996 [myid:] - INFO [Thread-12:FourLetterWordMain@43] - connecting to 127.0.0.1 11233
2017-01-24 22:12:57,997 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45986
2017-01-24 22:12:57,997 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:45986
2017-01-24 22:12:57,997 [myid:2] - INFO [Thread-165:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45986 (no session established for client)
2017-01-24 22:12:58,248 [myid:] - INFO [Thread-12:FourLetterWordMain@43] - connecting to 127.0.0.1 11233
2017-01-24 22:12:58,248 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45987
2017-01-24 22:12:58,249 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:45987
2017-01-24 22:12:58,249 [myid:2] - INFO [Thread-166:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45987 (no session established for client)
2017-01-24 22:12:58,499 [myid:] - INFO [Thread-12:FourLetterWordMain@43] - connecting to 127.0.0.1 11233
2017-01-24 22:12:58,500 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45988
2017-01-24 22:12:58,500 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:45988
2017-01-24 22:12:58,501 [myid:2] - INFO [Thread-167:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45988 (no session established for client)
2017-01-24 22:12:58,751 [myid:] - INFO [Thread-12:FourLetterWordMain@43] - connecting to 127.0.0.1 11233
2017-01-24 22:12:58,752 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45989
2017-01-24 22:12:58,752 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:45989
2017-01-24 22:12:58,752 [myid:2] - INFO [Thread-168:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45989 (no session established for client)
2017-01-24 22:12:58,983 [myid:] - WARN [Thread-12-SendThread(localhost:11233):ClientCnxn$SendThread@957] - SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: '/data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test2999887027410032136.junit.dir/jaas.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.
2017-01-24 22:12:58,983 [myid:] - INFO [Thread-12-SendThread(localhost:11233):ClientCnxn$SendThread@975] - Opening socket connection to server localhost/127.0.0.1:11233
2017-01-24 22:12:58,983 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45990
2017-01-24 22:12:58,983 [myid:] - INFO [Thread-12-SendThread(localhost:11233):ClientCnxn$SendThread@852] - Socket connection established, initiating session, client: /127.0.0.1:45990, server: localhost/127.0.0.1:11233
2017-01-24 22:12:58,984 [myid:2] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@354] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2017-01-24 22:12:58,984 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45990 (no session established for client)
2017-01-24 22:12:58,984 [myid:] - INFO [Thread-12-SendThread(localhost:11233):ClientCnxn$SendThread@1098] - Unable to read additional data from server sessionid 0x159d440f0ed0000, likely server has closed socket, closing socket connection and attempting reconnect
2017-01-24 22:12:59,003 [myid:] - INFO [Thread-12:FourLetterWordMain@43] - connecting to 127.0.0.1 11233
2017-01-24 22:12:59,003 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45991
2017-01-24 22:12:59,003 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:45991
2017-01-24 22:12:59,004 [myid:2] - INFO [Thread-169:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45991 (no session established for client)
2017-01-24 22:12:59,004 [myid:] - INFO [Thread-12:JUnit4ZKTestRunner$LoggedInvokeMethod@62] - TEST METHOD FAILED testRollingUpgrade
java.lang.AssertionError: waiting for server1being up
at org.junit.Assert.fail(Assert.java:91)
at org.junit.Assert.assertTrue(Assert.java:43)
at org.apache.zookeeper.server.quorum.auth.QuorumAuthUpgradeTest.restartServer(QuorumAuthUpgradeTest.java:232)
at org.apache.zookeeper.server.quorum.auth.QuorumAuthUpgradeTest.testRollingUpgrade(QuorumAuthUpgradeTest.java:204)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:52)
at org.junit.internal.runners.statements.FailOnTimeout$1.run(FailOnTimeout.java:28)
2017-01-24 22:12:59,005 [myid:] - INFO [main:QuorumBase@314] - Shutting down quorum peer QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11227
2017-01-24 22:12:59,005 [myid:] - INFO [main:Follower@167] - shutdown called
java.lang.Exception: shutdown Follower
at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:167)
at org.apache.zookeeper.server.quorum.QuorumPeer.shutdown(QuorumPeer.java:896)
at org.apache.zookeeper.test.QuorumBase.shutdown(QuorumBase.java:315)
at org.apache.zookeeper.server.quorum.QuorumPeerTestBase$TestQPMain.shutdown(QuorumPeerTestBase.java:59)
at org.apache.zookeeper.server.quorum.QuorumPeerTestBase$MainThread.shutdown(QuorumPeerTestBase.java:152)
at org.apache.zookeeper.server.quorum.auth.QuorumAuthTestBase.shutdown(QuorumAuthTestBase.java:138)
at org.apache.zookeeper.server.quorum.auth.QuorumAuthTestBase.shutdownAll(QuorumAuthTestBase.java:131)
at org.apache.zookeeper.server.quorum.auth.QuorumAuthUpgradeTest.tearDown(QuorumAuthUpgradeTest.java:68)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:37)
at org.junit.rules.TestWatchman$1.evaluate(TestWatchman.java:48)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:76)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
at org.junit.runners.ParentRunner.run(ParentRunner.java:236)
at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:39)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:535)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1182)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:1033)
2017-01-24 22:12:59,006 [myid:] - INFO [main:FollowerZooKeeperServer@139] - Shutting down
2017-01-24 22:12:59,006 [myid:] - INFO [main:ZooKeeperServer@419] - shutting down
2017-01-24 22:12:59,006 [myid:0] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11227:NIOServerCnxnFactory@224] - NIOServerCnxn factory exited run method
2017-01-24 22:12:59,007 [myid:0] - ERROR [localhost/127.0.0.1:11229:QuorumCnxManager$Listener@715] - Exception while listening
java.net.SocketException: Socket closed
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
at java.net.ServerSocket.implAccept(ServerSocket.java:530)
at java.net.ServerSocket.accept(ServerSocket.java:498)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$Listener.run(QuorumCnxManager.java:696)
2017-01-24 22:12:59,007 [myid:0] - WARN [SendWorker:2:QuorumCnxManager$SendWorker@896] - Interrupted while waiting for message on queue
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1049)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:73)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:884)
2017-01-24 22:12:59,007 [myid:0] - WARN [RecvWorker:2:QuorumCnxManager$RecvWorker@980] - Connection broken for id 2, my id = 0, error =
java.net.SocketException: Socket closed
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at java.io.BufferedInputStream.read(BufferedInputStream.java:254)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:965)
2017-01-24 22:12:59,008 [myid:0] - WARN [RecvWorker:2:QuorumCnxManager$RecvWorker@983] - Interrupting SendWorker
2017-01-24 22:12:59,008 [myid:0] - WARN [SendWorker:1:QuorumCnxManager$SendWorker@896] - Interrupted while waiting for message on queue
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1049)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:73)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:884)
2017-01-24 22:12:59,007 [myid:2] - WARN [RecvWorker:0:QuorumCnxManager$RecvWorker@980] - Connection broken for id 0, my id = 2, error =
java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:392)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:965)
2017-01-24 22:12:59,008 [myid:2] - WARN [RecvWorker:0:QuorumCnxManager$RecvWorker@983] - Interrupting SendWorker
2017-01-24 22:12:59,008 [myid:1] - WARN [RecvWorker:0:QuorumCnxManager$RecvWorker@980] - Connection broken for id 0, my id = 1, error =
java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:392)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:965)
2017-01-24 22:12:59,009 [myid:1] - WARN [RecvWorker:0:QuorumCnxManager$RecvWorker@983] - Interrupting SendWorker
2017-01-24 22:12:59,008 [myid:0] - WARN [RecvWorker:1:QuorumCnxManager$RecvWorker@980] - Connection broken for id 1, my id = 0, error =
java.net.SocketException: Socket closed
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at java.io.BufferedInputStream.read(BufferedInputStream.java:254)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:965)
2017-01-24 22:12:59,009 [myid:0] - WARN [RecvWorker:1:QuorumCnxManager$RecvWorker@983] - Interrupting SendWorker
2017-01-24 22:12:59,008 [myid:0] - WARN [SendWorker:2:QuorumCnxManager$SendWorker@906] - Send worker leaving thread
2017-01-24 22:12:59,010 [myid:1] - WARN [SendWorker:0:QuorumCnxManager$SendWorker@896] - Interrupted while waiting for message on queue
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1049)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:73)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:884)
2017-01-24 22:12:59,009 [myid:0] - WARN [SendWorker:1:QuorumCnxManager$SendWorker@906] - Send worker leaving thread
2017-01-24 22:12:59,010 [myid:2] - WARN [SendWorker:0:QuorumCnxManager$SendWorker@896] - Interrupted while waiting for message on queue
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1049)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:73)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:884)
2017-01-24 22:12:59,011 [myid:1] - WARN [SendWorker:0:QuorumCnxManager$SendWorker@906] - Send worker leaving thread
2017-01-24 22:12:59,011 [myid:2] - WARN [SendWorker:0:QuorumCnxManager$SendWorker@906] - Send worker leaving thread
2017-01-24 22:12:59,011 [myid:] - INFO [main:QuorumBase@318] - Shutting down leader election QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11227
2017-01-24 22:12:59,011 [myid:] - INFO [main:QuorumBase@323] - Waiting for QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11227 to exit thread
2017-01-24 22:12:59,371 [myid:0] - INFO [WorkerSender[myid=0]:FastLeaderElection$Messenger$WorkerSender@370] - WorkerSender is down
2017-01-24 22:12:59,372 [myid:0] - INFO [WorkerReceiver[myid=0]:FastLeaderElection$Messenger$WorkerReceiver@340] - WorkerReceiver is down
2017-01-24 22:12:59,885 [myid:] - WARN [Thread-12-SendThread(localhost:11227):ClientCnxn$SendThread@957] - SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: '/data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test2999887027410032136.junit.dir/jaas.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.
2017-01-24 22:12:59,886 [myid:] - INFO [Thread-12-SendThread(localhost:11227):ClientCnxn$SendThread@975] - Opening socket connection to server localhost/127.0.0.1:11227
2017-01-24 22:12:59,886 [myid:] - WARN [Thread-12-SendThread(localhost:11227):ClientCnxn$SendThread@1102] - Session 0x159d440f0ed0000 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
2017-01-24 22:13:00,007 [myid:0] - INFO [localhost/127.0.0.1:11229:QuorumCnxManager$Listener@728] - Leaving listener
2017-01-24 22:13:01,546 [myid:] - WARN [Thread-12-SendThread(localhost:11230):ClientCnxn$SendThread@957] - SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: '/data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test2999887027410032136.junit.dir/jaas.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.
2017-01-24 22:13:01,546 [myid:] - INFO [Thread-12-SendThread(localhost:11230):ClientCnxn$SendThread@975] - Opening socket connection to server localhost/127.0.0.1:11230
2017-01-24 22:13:01,547 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11230:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:52524
2017-01-24 22:13:01,547 [myid:] - INFO [Thread-12-SendThread(localhost:11230):ClientCnxn$SendThread@852] - Socket connection established, initiating session, client: /127.0.0.1:52524, server: localhost/127.0.0.1:11230
2017-01-24 22:13:01,547 [myid:1] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11230:NIOServerCnxn@354] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2017-01-24 22:13:01,547 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11230:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:52524 (no session established for client)
2017-01-24 22:13:01,547 [myid:] - INFO [Thread-12-SendThread(localhost:11230):ClientCnxn$SendThread@1098] - Unable to read additional data from server sessionid 0x159d440f0ed0000, likely server has closed socket, closing socket connection and attempting reconnect
2017-01-24 22:13:01,960 [myid:] - WARN [Thread-12-SendThread(localhost:11233):ClientCnxn$SendThread@957] - SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: '/data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test2999887027410032136.junit.dir/jaas.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.
2017-01-24 22:13:01,960 [myid:] - INFO [Thread-12-SendThread(localhost:11233):ClientCnxn$SendThread@975] - Opening socket connection to server localhost/127.0.0.1:11233
2017-01-24 22:13:01,961 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45994
2017-01-24 22:13:01,961 [myid:] - INFO [Thread-12-SendThread(localhost:11233):ClientCnxn$SendThread@852] - Socket connection established, initiating session, client: /127.0.0.1:45994, server: localhost/127.0.0.1:11233
2017-01-24 22:13:01,961 [myid:2] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@354] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2017-01-24 22:13:01,961 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45994 (no session established for client)
2017-01-24 22:13:01,961 [myid:] - INFO [Thread-12-SendThread(localhost:11233):ClientCnxn$SendThread@1098] - Unable to read additional data from server sessionid 0x159d440f0ed0000, likely server has closed socket, closing socket connection and attempting reconnect
2017-01-24 22:13:03,003 [myid:] - WARN [Thread-12-SendThread(localhost:11227):ClientCnxn$SendThread@957] - SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: '/data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test2999887027410032136.junit.dir/jaas.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.
2017-01-24 22:13:03,003 [myid:] - INFO [Thread-12-SendThread(localhost:11227):ClientCnxn$SendThread@975] - Opening socket connection to server localhost/127.0.0.1:11227
2017-01-24 22:13:03,004 [myid:] - WARN [Thread-12-SendThread(localhost:11227):ClientCnxn$SendThread@1102] - Session 0x159d440f0ed0000 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
2017-01-24 22:13:04,254 [myid:] - WARN [Thread-12-SendThread(localhost:11230):ClientCnxn$SendThread@957] - SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: '/data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test2999887027410032136.junit.dir/jaas.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.
2017-01-24 22:13:04,254 [myid:] - INFO [Thread-12-SendThread(localhost:11230):ClientCnxn$SendThread@975] - Opening socket connection to server localhost/127.0.0.1:11230
2017-01-24 22:13:04,254 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11230:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:52527
2017-01-24 22:13:04,254 [myid:] - INFO [Thread-12-SendThread(localhost:11230):ClientCnxn$SendThread@852] - Socket connection established, initiating session, client: /127.0.0.1:52527, server: localhost/127.0.0.1:11230
2017-01-24 22:13:04,255 [myid:1] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11230:NIOServerCnxn@354] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2017-01-24 22:13:04,255 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11230:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:52527 (no session established for client)
2017-01-24 22:13:04,255 [myid:] - INFO [Thread-12-SendThread(localhost:11230):ClientCnxn$SendThread@1098] - Unable to read additional data from server sessionid 0x159d440f0ed0000, likely server has closed socket, closing socket connection and attempting reconnect
2017-01-24 22:13:04,947 [myid:] - WARN [Thread-12-SendThread(localhost:11233):ClientCnxn$SendThread@957] - SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: '/data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test2999887027410032136.junit.dir/jaas.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.
2017-01-24 22:13:04,947 [myid:] - INFO [Thread-12-SendThread(localhost:11233):ClientCnxn$SendThread@975] - Opening socket connection to server localhost/127.0.0.1:11233
2017-01-24 22:13:04,947 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:45997
2017-01-24 22:13:04,947 [myid:] - INFO [Thread-12-SendThread(localhost:11233):ClientCnxn$SendThread@852] - Socket connection established, initiating session, client: /127.0.0.1:45997, server: localhost/127.0.0.1:11233
2017-01-24 22:13:04,948 [myid:2] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@354] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2017-01-24 22:13:04,948 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:45997 (no session established for client)
2017-01-24 22:13:04,948 [myid:] - INFO [Thread-12-SendThread(localhost:11233):ClientCnxn$SendThread@1098] - Unable to read additional data from server sessionid 0x159d440f0ed0000, likely server has closed socket, closing socket connection and attempting reconnect
2017-01-24 22:13:05,888 [myid:] - WARN [Thread-12-SendThread(localhost:11227):ClientCnxn$SendThread@957] - SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: '/data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test2999887027410032136.junit.dir/jaas.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.
2017-01-24 22:13:05,889 [myid:] - INFO [Thread-12-SendThread(localhost:11227):ClientCnxn$SendThread@975] - Opening socket connection to server localhost/127.0.0.1:11227
2017-01-24 22:13:05,889 [myid:] - WARN [Thread-12-SendThread(localhost:11227):ClientCnxn$SendThread@1102] - Session 0x159d440f0ed0000 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
2017-01-24 22:13:07,549 [myid:] - WARN [Thread-12-SendThread(localhost:11230):ClientCnxn$SendThread@957] - SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: '/data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test2999887027410032136.junit.dir/jaas.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.
2017-01-24 22:13:07,549 [myid:] - INFO [Thread-12-SendThread(localhost:11230):ClientCnxn$SendThread@975] - Opening socket connection to server localhost/127.0.0.1:11230
2017-01-24 22:13:07,550 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11230:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:52530
2017-01-24 22:13:07,550 [myid:] - INFO [Thread-12-SendThread(localhost:11230):ClientCnxn$SendThread@852] - Socket connection established, initiating session, client: /127.0.0.1:52530, server: localhost/127.0.0.1:11230
2017-01-24 22:13:07,550 [myid:1] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11230:NIOServerCnxn@354] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2017-01-24 22:13:07,550 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11230:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:52530 (no session established for client)
2017-01-24 22:13:07,550 [myid:] - INFO [Thread-12-SendThread(localhost:11230):ClientCnxn$SendThread@1098] - Unable to read additional data from server sessionid 0x159d440f0ed0000, likely server has closed socket, closing socket connection and attempting reconnect
2017-01-24 22:13:07,986 [myid:] - WARN [Thread-12-SendThread(localhost:11233):ClientCnxn$SendThread@957] - SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: '/data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test2999887027410032136.junit.dir/jaas.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.
2017-01-24 22:13:07,986 [myid:] - INFO [Thread-12-SendThread(localhost:11233):ClientCnxn$SendThread@975] - Opening socket connection to server localhost/127.0.0.1:11233
2017-01-24 22:13:07,987 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:46000
2017-01-24 22:13:07,987 [myid:] - INFO [Thread-12-SendThread(localhost:11233):ClientCnxn$SendThread@852] - Socket connection established, initiating session, client: /127.0.0.1:46000, server: localhost/127.0.0.1:11233
2017-01-24 22:13:07,987 [myid:2] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@354] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2017-01-24 22:13:07,987 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:46000 (no session established for client)
2017-01-24 22:13:07,987 [myid:] - INFO [Thread-12-SendThread(localhost:11233):ClientCnxn$SendThread@1098] - Unable to read additional data from server sessionid 0x159d440f0ed0000, likely server has closed socket, closing socket connection and attempting reconnect
2017-01-24 22:13:08,583 [myid:] - WARN [Thread-12-SendThread(localhost:11227):ClientCnxn$SendThread@957] - SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: '/data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test2999887027410032136.junit.dir/jaas.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.
2017-01-24 22:13:08,583 [myid:] - INFO [Thread-12-SendThread(localhost:11227):ClientCnxn$SendThread@975] - Opening socket connection to server localhost/127.0.0.1:11227
2017-01-24 22:13:08,583 [myid:] - WARN [Thread-12-SendThread(localhost:11227):ClientCnxn$SendThread@1102] - Session 0x159d440f0ed0000 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
2017-01-24 22:13:08,866 [myid:2] - WARN [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11233:QuorumPeer@867] - Unexpected exception
java.lang.InterruptedException: Timeout while waiting for epoch to be acked by quorum
at org.apache.zookeeper.server.quorum.Leader.waitForEpochAck(Leader.java:906)
at org.apache.zookeeper.server.quorum.Leader.lead(Leader.java:392)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:864)
2017-01-24 22:13:08,866 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11233:Leader@491] - Shutting down
2017-01-24 22:13:08,866 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11233:Leader@497] - Shutdown called
java.lang.Exception: shutdown Leader! reason: Forcing shutdown
at org.apache.zookeeper.server.quorum.Leader.shutdown(Leader.java:497)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:870)
2017-01-24 22:13:08,867 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11233:ZooKeeperServer@419] - shutting down
2017-01-24 22:13:08,867 [myid:2] - INFO [Thread-49:Leader$LearnerCnxAcceptor@318] - exception while shutting down acceptor: java.net.SocketException: Socket closed
2017-01-24 22:13:08,867 [myid:0] - WARN [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11227:Follower@90] - Exception when following the leader
java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:392)
at org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63)
at org.apache.zookeeper.server.quorum.QuorumPacket.deserialize(QuorumPacket.java:83)
at org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:99)
at org.apache.zookeeper.server.quorum.Learner.readPacket(Learner.java:152)
at org.apache.zookeeper.server.quorum.Learner.syncWithLeader(Learner.java:321)
at org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:83)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:851)
2017-01-24 22:13:08,867 [myid:2] - ERROR [LearnerHandler-/127.0.0.1:42608:LearnerHandler@596] - Unexpected exception causing shutdown
java.lang.InterruptedException
at java.lang.Object.wait(Native Method)
at org.apache.zookeeper.server.quorum.Leader.waitForEpochAck(Leader.java:902)
at org.apache.zookeeper.server.quorum.LearnerHandler.run(LearnerHandler.java:322)
2017-01-24 22:13:08,868 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11227:Follower@167] - shutdown called
java.lang.Exception: shutdown Follower
at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:167)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:855)
2017-01-24 22:13:08,868 [myid:2] - ERROR [LearnerHandler-/127.0.0.1:42602:LearnerHandler@596] - Unexpected exception causing shutdown
java.lang.InterruptedException
at java.lang.Object.wait(Native Method)
at org.apache.zookeeper.server.quorum.Leader.waitForEpochAck(Leader.java:902)
at org.apache.zookeeper.server.quorum.LearnerHandler.run(LearnerHandler.java:322)
2017-01-24 22:13:08,868 [myid:1] - WARN [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11230:Follower@90] - Exception when following the leader
java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:392)
at org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63)
at org.apache.zookeeper.server.quorum.QuorumPacket.deserialize(QuorumPacket.java:83)
at org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:99)
at org.apache.zookeeper.server.quorum.Learner.readPacket(Learner.java:152)
at org.apache.zookeeper.server.quorum.Learner.syncWithLeader(Learner.java:321)
at org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:83)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:851)
2017-01-24 22:13:08,868 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11233:QuorumPeer@781] - LOOKING
2017-01-24 22:13:08,869 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11230:Follower@167] - shutdown called
java.lang.Exception: shutdown Follower
at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:167)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:855)
2017-01-24 22:13:08,869 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11230:FollowerZooKeeperServer@139] - Shutting down
2017-01-24 22:13:08,869 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11230:ZooKeeperServer@419] - shutting down
2017-01-24 22:13:08,870 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11230:QuorumPeer@781] - LOOKING
2017-01-24 22:13:08,868 [myid:2] - WARN [LearnerHandler-/127.0.0.1:42602:LearnerHandler@598] - ******* GOODBYE /127.0.0.1:42602 ********
2017-01-24 22:13:08,870 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11230:FileSnap@83] - Reading snapshot /data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test4772652757083079773.junit.dir/data/version-2/snapshot.200000002
2017-01-24 22:13:08,871 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11230:FastLeaderElection@744] - New election. My id = 1, proposed zxid=0x300000001
2017-01-24 22:13:08,868 [myid:2] - WARN [LearnerHandler-/127.0.0.1:42608:LearnerHandler@598] - ******* GOODBYE /127.0.0.1:42608 ********
2017-01-24 22:13:08,868 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11227:FollowerZooKeeperServer@139] - Shutting down
2017-01-24 22:13:08,872 [myid:1] - WARN [WorkerSender[myid=1]:QuorumCnxManager@559] - Cannot open channel to 0 at election address localhost/127.0.0.1:11229
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:538)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:514)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:393)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:365)
at java.lang.Thread.run(Thread.java:745)
2017-01-24 22:13:08,872 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11227:ZooKeeperServer@419] - shutting down
2017-01-24 22:13:08,872 [myid:0] - WARN [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11227:QuorumPeer@879] - QuorumPeer main thread exited
2017-01-24 22:13:08,873 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@542] - Notification: 1 (n.leader), 0x300000001 (n.zxid), 0x4 (n.round), LOOKING (n.state), 1 (n.sid), 0x3 (n.peerEPoch), LOOKING (my state)
2017-01-24 22:13:08,870 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11233:FileSnap@83] - Reading snapshot /data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test6343054086158567637.junit.dir/data/version-2/snapshot.0
2017-01-24 22:13:08,874 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@542] - Notification: 1 (n.leader), 0x300000001 (n.zxid), 0x4 (n.round), LOOKING (n.state), 1 (n.sid), 0x3 (n.peerEPoch), LOOKING (my state)
2017-01-24 22:13:08,875 [myid:] - INFO [main:QuorumBase@314] - Shutting down quorum peer QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11233
2017-01-24 22:13:08,875 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11233:NIOServerCnxnFactory@224] - NIOServerCnxn factory exited run method
2017-01-24 22:13:08,875 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11233:FastLeaderElection@744] - New election. My id = 2, proposed zxid=0x0
2017-01-24 22:13:08,876 [myid:2] - WARN [WorkerSender[myid=2]:QuorumCnxManager@559] - Cannot open channel to 0 at election address localhost/127.0.0.1:11229
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:538)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:514)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:393)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:365)
at java.lang.Thread.run(Thread.java:745)
2017-01-24 22:13:08,877 [myid:2] - INFO [WorkerSender[myid=2]:FastLeaderElection$Messenger$WorkerSender@370] - WorkerSender is down
2017-01-24 22:13:08,876 [myid:2] - WARN [SendWorker:1:QuorumCnxManager$SendWorker@896] - Interrupted while waiting for message on queue
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1049)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:73)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:884)
2017-01-24 22:13:08,877 [myid:2] - WARN [SendWorker:1:QuorumCnxManager$SendWorker@906] - Send worker leaving thread
2017-01-24 22:13:08,877 [myid:2] - WARN [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11233:QuorumPeer@829] - Unexpected exception
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:758)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:827)
2017-01-24 22:13:08,878 [myid:2] - WARN [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11233:QuorumPeer@879] - QuorumPeer main thread exited
2017-01-24 22:13:08,877 [myid:] - INFO [main:QuorumBase@318] - Shutting down leader election QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11233
2017-01-24 22:13:08,878 [myid:] - INFO [main:QuorumBase@323] - Waiting for QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11233 to exit thread
2017-01-24 22:13:08,877 [myid:2] - WARN [RecvWorker:1:QuorumCnxManager$RecvWorker@980] - Connection broken for id 1, my id = 2, error =
java.net.SocketException: Socket closed
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at java.io.BufferedInputStream.read(BufferedInputStream.java:254)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:965)
2017-01-24 22:13:08,879 [myid:2] - WARN [RecvWorker:1:QuorumCnxManager$RecvWorker@983] - Interrupting SendWorker
2017-01-24 22:13:08,876 [myid:1] - WARN [RecvWorker:2:QuorumCnxManager$RecvWorker@980] - Connection broken for id 2, my id = 1, error =
java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:392)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:965)
2017-01-24 22:13:08,879 [myid:1] - WARN [RecvWorker:2:QuorumCnxManager$RecvWorker@983] - Interrupting SendWorker
2017-01-24 22:13:08,876 [myid:2] - ERROR [localhost/127.0.0.1:11235:QuorumCnxManager$Listener@715] - Exception while listening
java.net.SocketException: Socket closed
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
at java.net.ServerSocket.implAccept(ServerSocket.java:530)
at java.net.ServerSocket.accept(ServerSocket.java:498)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$Listener.run(QuorumCnxManager.java:696)
2017-01-24 22:13:08,880 [myid:1] - WARN [SendWorker:2:QuorumCnxManager$SendWorker@896] - Interrupted while waiting for message on queue
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1049)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:73)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:884)
2017-01-24 22:13:08,880 [myid:1] - WARN [SendWorker:2:QuorumCnxManager$SendWorker@906] - Send worker leaving thread
2017-01-24 22:13:08,880 [myid:] - INFO [main:ZKTestCase$1@65] - FAILED testRollingUpgrade
java.lang.AssertionError: waiting for server1being up
at org.junit.Assert.fail(Assert.java:91)
at org.junit.Assert.assertTrue(Assert.java:43)
at org.apache.zookeeper.server.quorum.auth.QuorumAuthUpgradeTest.restartServer(QuorumAuthUpgradeTest.java:232)
at org.apache.zookeeper.server.quorum.auth.QuorumAuthUpgradeTest.testRollingUpgrade(QuorumAuthUpgradeTest.java:204)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:52)
at org.junit.internal.runners.statements.FailOnTimeout$1.run(FailOnTimeout.java:28)
2017-01-24 22:13:08,880 [myid:] - INFO [main:ZKTestCase$1@55] - FINISHED testRollingUpgrade
2017-01-24 22:13:08,885 [myid:] - INFO [main:ZKTestCase$1@50] - STARTING testNullAuthLearnerServer
2017-01-24 22:13:08,885 [myid:] - INFO [Thread-170:JUnit4ZKTestRunner$LoggedInvokeMethod@50] - RUNNING TEST METHOD testNullAuthLearnerServer
2017-01-24 22:13:08,885 [myid:] - INFO [Thread-170:PortAssignment@32] - assigning port 11236
2017-01-24 22:13:08,886 [myid:] - INFO [Thread-170:PortAssignment@32] - assigning port 11237
2017-01-24 22:13:08,886 [myid:] - INFO [Thread-170:PortAssignment@32] - assigning port 11238
2017-01-24 22:13:08,886 [myid:] - INFO [Thread-170:PortAssignment@32] - assigning port 11239
2017-01-24 22:13:08,886 [myid:] - INFO [Thread-170:PortAssignment@32] - assigning port 11240
2017-01-24 22:13:08,886 [myid:] - INFO [Thread-170:PortAssignment@32] - assigning port 11241
2017-01-24 22:13:08,886 [myid:] - INFO [Thread-170:QuorumPeerTestBase$MainThread@81] - id = 0 tmpDir = /data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test6384659498769109567.junit.dir clientPort = 11236
2017-01-24 22:13:08,887 [myid:] - INFO [Thread-171:QuorumPeerConfig@111] - Reading configuration from: /data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test6384659498769109567.junit.dir/zoo.cfg
2017-01-24 22:13:08,887 [myid:] - INFO [Thread-170:QuorumPeerTestBase$MainThread@81] - id = 1 tmpDir = /data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test5470635973333042784.junit.dir clientPort = 11239
2017-01-24 22:13:08,887 [myid:] - WARN [Thread-171:QuorumPeerConfig@327] - No server failure will be tolerated. You need at least 3 servers.
2017-01-24 22:13:08,888 [myid:] - INFO [Thread-171:QuorumPeerConfig@374] - Defaulting to majority quorums
2017-01-24 22:13:08,888 [myid:0] - INFO [Thread-171:DatadirCleanupManager@78] - autopurge.snapRetainCount set to 3
2017-01-24 22:13:08,888 [myid:0] - INFO [Thread-171:DatadirCleanupManager@79] - autopurge.purgeInterval set to 0
2017-01-24 22:13:08,888 [myid:0] - INFO [Thread-171:DatadirCleanupManager@101] - Purge task is not scheduled.
2017-01-24 22:13:08,888 [myid:0] - WARN [Thread-171:QuorumPeerMain@129] - Unable to register log4j JMX control
javax.management.InstanceAlreadyExistsException: log4j:hiearchy=default
at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:437)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1898)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:966)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324)
at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
at org.apache.zookeeper.jmx.ManagedUtil.registerLog4jMBeans(ManagedUtil.java:53)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:127)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:116)
at org.apache.zookeeper.server.quorum.QuorumPeerTestBase$MainThread.run(QuorumPeerTestBase.java:140)
at java.lang.Thread.run(Thread.java:745)
2017-01-24 22:13:08,888 [myid:0] - INFO [Thread-171:QuorumPeerMain@132] - Starting quorum peer
2017-01-24 22:13:08,889 [myid:] - INFO [Thread-170:FourLetterWordMain@43] - connecting to 127.0.0.1 11236
2017-01-24 22:13:08,897 [myid:] - INFO [Thread-170:ClientBase@246] - server 127.0.0.1:11236 not up java.net.ConnectException: Connection refused
2017-01-24 22:13:08,889 [myid:] - INFO [Thread-172:QuorumPeerConfig@111] - Reading configuration from: /data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test5470635973333042784.junit.dir/zoo.cfg
2017-01-24 22:13:08,897 [myid:] - WARN [Thread-172:QuorumPeerConfig@327] - No server failure will be tolerated. You need at least 3 servers.
2017-01-24 22:13:08,897 [myid:] - INFO [Thread-172:QuorumPeerConfig@374] - Defaulting to majority quorums
2017-01-24 22:13:08,897 [myid:1] - INFO [Thread-172:DatadirCleanupManager@78] - autopurge.snapRetainCount set to 3
2017-01-24 22:13:08,898 [myid:1] - INFO [Thread-172:DatadirCleanupManager@79] - autopurge.purgeInterval set to 0
2017-01-24 22:13:08,898 [myid:1] - INFO [Thread-172:DatadirCleanupManager@101] - Purge task is not scheduled.
2017-01-24 22:13:08,898 [myid:1] - WARN [Thread-172:QuorumPeerMain@129] - Unable to register log4j JMX control
javax.management.InstanceAlreadyExistsException: log4j:hiearchy=default
at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:437)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1898)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:966)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324)
at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
at org.apache.zookeeper.jmx.ManagedUtil.registerLog4jMBeans(ManagedUtil.java:53)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:127)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:116)
at org.apache.zookeeper.server.quorum.QuorumPeerTestBase$MainThread.run(QuorumPeerTestBase.java:140)
at java.lang.Thread.run(Thread.java:745)
2017-01-24 22:13:08,898 [myid:1] - INFO [Thread-172:QuorumPeerMain@132] - Starting quorum peer
2017-01-24 22:13:08,899 [myid:1] - INFO [Thread-172:NIOServerCnxnFactory@94] - binding to port 0.0.0.0/0.0.0.0:11239
2017-01-24 22:13:08,899 [myid:1] - INFO [Thread-172:QuorumPeer@1048] - minSessionTimeout set to -1
2017-01-24 22:13:08,899 [myid:1] - INFO [Thread-172:QuorumPeer@1059] - maxSessionTimeout set to -1
2017-01-24 22:13:08,900 [myid:1] - INFO [Thread-172:QuorumPeer@1277] - QuorumPeer communication is not secured!
2017-01-24 22:13:08,900 [myid:1] - INFO [Thread-172:QuorumPeer@1306] - quorum.cnxn.threads.size set to 20
2017-01-24 22:13:08,900 [myid:1] - INFO [Thread-172:QuorumPeer@540] - currentEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
2017-01-24 22:13:08,889 [myid:0] - INFO [Thread-171:NIOServerCnxnFactory@94] - binding to port 0.0.0.0/0.0.0.0:11236
2017-01-24 22:13:08,901 [myid:0] - INFO [Thread-171:QuorumPeer@1048] - minSessionTimeout set to -1
2017-01-24 22:13:08,901 [myid:0] - INFO [Thread-171:QuorumPeer@1059] - maxSessionTimeout set to -1
2017-01-24 22:13:08,901 [myid:0] - INFO [Thread-171:QuorumPeer@1277] - QuorumPeer communication is not secured!
2017-01-24 22:13:08,901 [myid:0] - INFO [Thread-171:QuorumPeer@1306] - quorum.cnxn.threads.size set to 20
2017-01-24 22:13:08,902 [myid:0] - INFO [Thread-171:QuorumPeer@540] - currentEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
2017-01-24 22:13:08,903 [myid:1] - INFO [Thread-172:QuorumPeer@555] - acceptedEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
2017-01-24 22:13:08,903 [myid:0] - INFO [Thread-171:QuorumPeer@555] - acceptedEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
2017-01-24 22:13:08,905 [myid:1] - INFO [Thread-173:QuorumCnxManager$Listener@691] - My election bind port: 0.0.0.0/0.0.0.0:11241
2017-01-24 22:13:08,909 [myid:0] - INFO [Thread-174:QuorumCnxManager$Listener@691] - My election bind port: 0.0.0.0/0.0.0.0:11238
2017-01-24 22:13:08,910 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11236:QuorumPeer@781] - LOOKING
2017-01-24 22:13:08,910 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11239:QuorumPeer@781] - LOOKING
2017-01-24 22:13:08,910 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11236:FastLeaderElection@744] - New election. My id = 0, proposed zxid=0x0
2017-01-24 22:13:08,910 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11239:FastLeaderElection@744] - New election. My id = 1, proposed zxid=0x0
2017-01-24 22:13:08,911 [myid:0] - INFO [WorkerSender[myid=0]:QuorumCnxManager@331] - Have smaller server identifier, so dropping the connection: (1, 0)
2017-01-24 22:13:08,911 [myid:0] - INFO [localhost/127.0.0.1:11238:QuorumCnxManager$Listener@698] - Received connection request /127.0.0.1:59130
2017-01-24 22:13:08,911 [myid:1] - INFO [localhost/127.0.0.1:11241:QuorumCnxManager$Listener@698] - Received connection request /127.0.0.1:51284
2017-01-24 22:13:08,911 [myid:0] - INFO [WorkerReceiver[myid=0]:FastLeaderElection@542] - Notification: 0 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 0 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)
2017-01-24 22:13:08,912 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@542] - Notification: 1 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)
2017-01-24 22:13:08,912 [myid:1] - WARN [SendWorker:0:QuorumCnxManager$SendWorker@901] - Exception when using channel: for id 0 my id = 1 error = java.net.SocketException: Socket closed
2017-01-24 22:13:08,913 [myid:1] - WARN [SendWorker:0:QuorumCnxManager$SendWorker@906] - Send worker leaving thread
2017-01-24 22:13:08,913 [myid:1] - WARN [RecvWorker:0:QuorumCnxManager$RecvWorker@983] - Interrupting SendWorker
2017-01-24 22:13:08,914 [myid:0] - WARN [SendWorker:1:QuorumCnxManager$SendWorker@901] - Exception when using channel: for id 1 my id = 0 error = java.net.SocketException: Broken pipe
2017-01-24 22:13:08,915 [myid:0] - INFO [localhost/127.0.0.1:11238:QuorumCnxManager$Listener@698] - Received connection request /127.0.0.1:59132
2017-01-24 22:13:08,915 [myid:0] - WARN [RecvWorker:1:QuorumCnxManager$RecvWorker@983] - Interrupting SendWorker
2017-01-24 22:13:08,915 [myid:0] - WARN [SendWorker:1:QuorumCnxManager$SendWorker@906] - Send worker leaving thread
2017-01-24 22:13:08,917 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@542] - Notification: 0 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 0 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)
2017-01-24 22:13:08,917 [myid:0] - INFO [WorkerReceiver[myid=0]:FastLeaderElection@542] - Notification: 1 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)
2017-01-24 22:13:08,918 [myid:0] - INFO [WorkerReceiver[myid=0]:FastLeaderElection@542] - Notification: 1 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 0 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)
2017-01-24 22:13:08,921 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@542] - Notification: 1 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 0 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)
2017-01-24 22:13:09,074 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11230:FastLeaderElection@778] - Notification time out: 400
2017-01-24 22:13:09,075 [myid:1] - WARN [WorkerSender[myid=1]:QuorumCnxManager@559] - Cannot open channel to 0 at election address localhost/127.0.0.1:11229
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:538)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:514)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:393)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:365)
at java.lang.Thread.run(Thread.java:745)
2017-01-24 22:13:09,075 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@542] - Notification: 1 (n.leader), 0x300000001 (n.zxid), 0x4 (n.round), LOOKING (n.state), 1 (n.sid), 0x3 (n.peerEPoch), LOOKING (my state)
2017-01-24 22:13:09,075 [myid:1] - WARN [WorkerSender[myid=1]:QuorumCnxManager@559] - Cannot open channel to 2 at election address localhost/127.0.0.1:11235
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:538)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:514)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:393)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:365)
at java.lang.Thread.run(Thread.java:745)
2017-01-24 22:13:09,119 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11236:QuorumPeer@849] - FOLLOWING
2017-01-24 22:13:09,119 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11236:ZooKeeperServer@162] - Created server with tickTime 4000 minSessionTimeout 8000 maxSessionTimeout 80000 datadir /data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test6384659498769109567.junit.dir/data/version-2 snapdir /data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test6384659498769109567.junit.dir/data/version-2
2017-01-24 22:13:09,119 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11236:Follower@64] - FOLLOWING - LEADER ELECTION TOOK - 209
2017-01-24 22:13:09,120 [myid:0] - WARN [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11236:Learner@233] - Unexpected exception, tries=0, connecting to localhost/127.0.0.1:11240
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.server.quorum.Learner.connectToLeader(Learner.java:225)
at org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:72)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:851)
2017-01-24 22:13:09,122 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11239:QuorumPeer@861] - LEADING
2017-01-24 22:13:09,122 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11239:ZooKeeperServer@162] - Created server with tickTime 4000 minSessionTimeout 8000 maxSessionTimeout 80000 datadir /data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test5470635973333042784.junit.dir/data/version-2 snapdir /data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test5470635973333042784.junit.dir/data/version-2
2017-01-24 22:13:09,122 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11239:Leader@356] - LEADING - LEADER ELECTION TOOK - 212
2017-01-24 22:13:09,123 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11239:FileTxnSnapLog@281] - Snapshotting: 0x0 to /data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test5470635973333042784.junit.dir/data/version-2/snapshot.0
2017-01-24 22:13:09,147 [myid:] - INFO [Thread-170:FourLetterWordMain@43] - connecting to 127.0.0.1 11236
2017-01-24 22:13:09,147 [myid:0] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11236:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:40329
2017-01-24 22:13:09,147 [myid:0] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11236:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:40329
2017-01-24 22:13:09,148 [myid:0] - INFO [Thread-176:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:40329 (no session established for client)
2017-01-24 22:13:09,399 [myid:] - INFO [Thread-170:FourLetterWordMain@43] - connecting to 127.0.0.1 11236
2017-01-24 22:13:09,399 [myid:0] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11236:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:40330
2017-01-24 22:13:09,399 [myid:0] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11236:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:40330
2017-01-24 22:13:09,400 [myid:0] - INFO [Thread-177:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:40330 (no session established for client)
2017-01-24 22:13:09,476 [myid:1] - WARN [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11230:QuorumCnxManager@559] - Cannot open channel to 0 at election address localhost/127.0.0.1:11229
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:538)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:579)
at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:769)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:827)
2017-01-24 22:13:09,477 [myid:1] - WARN [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11230:QuorumCnxManager@559] - Cannot open channel to 2 at election address localhost/127.0.0.1:11235
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:538)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:579)
at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:769)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:827)
2017-01-24 22:13:09,477 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11230:FastLeaderElection@778] - Notification time out: 800
2017-01-24 22:13:09,650 [myid:] - INFO [Thread-170:FourLetterWordMain@43] - connecting to 127.0.0.1 11236
2017-01-24 22:13:09,651 [myid:0] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11236:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:40333
2017-01-24 22:13:09,651 [myid:0] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11236:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:40333
2017-01-24 22:13:09,651 [myid:0] - INFO [Thread-178:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:40333 (no session established for client)
2017-01-24 22:13:09,880 [myid:2] - INFO [localhost/127.0.0.1:11235:QuorumCnxManager$Listener@728] - Leaving listener
2017-01-24 22:13:09,902 [myid:] - INFO [Thread-170:FourLetterWordMain@43] - connecting to 127.0.0.1 11236
2017-01-24 22:13:09,902 [myid:0] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11236:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:40334
2017-01-24 22:13:09,903 [myid:0] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11236:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:40334
2017-01-24 22:13:09,903 [myid:0] - INFO [Thread-179:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:40334 (no session established for client)
2017-01-24 22:13:10,121 [myid:1] - INFO [LearnerHandler-/127.0.0.1:51982:LearnerHandler@287] - Follower sid: 0 : info : org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@36d2499a
2017-01-24 22:13:10,123 [myid:1] - INFO [LearnerHandler-/127.0.0.1:51982:LearnerHandler@342] - Synchronizing with Follower sid: 0 maxCommittedLog=0x0 minCommittedLog=0x0 peerLastZxid=0x0
2017-01-24 22:13:10,124 [myid:1] - INFO [LearnerHandler-/127.0.0.1:51982:LearnerHandler@441] - Sending snapshot last zxid of peer is 0x0 zxid of leader is 0x100000000sent zxid of db as 0x0
2017-01-24 22:13:10,124 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11236:Learner@329] - Getting a snapshot from leader
2017-01-24 22:13:10,125 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11236:FileTxnSnapLog@281] - Snapshotting: 0x0 to /data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test6384659498769109567.junit.dir/data/version-2/snapshot.0
2017-01-24 22:13:10,126 [myid:1] - INFO [LearnerHandler-/127.0.0.1:51982:LearnerHandler@477] - Received NEWLEADER-ACK message from 0
2017-01-24 22:13:10,126 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11239:Leader@934] - Have quorum of supporters, sids: [ 0,1 ]; starting up and setting last processed zxid: 0x100000000
2017-01-24 22:13:10,153 [myid:] - INFO [Thread-170:FourLetterWordMain@43] - connecting to 127.0.0.1 11236
2017-01-24 22:13:10,154 [myid:0] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11236:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:40336
2017-01-24 22:13:10,154 [myid:0] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11236:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:40336
2017-01-24 22:13:10,155 [myid:0] - INFO [Thread-181:NIOServerCnxn$StatCommand@655] - Stat command output
2017-01-24 22:13:10,155 [myid:0] - INFO [Thread-181:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:40336 (no session established for client)
2017-01-24 22:13:10,155 [myid:] - INFO [Thread-170:FourLetterWordMain@43] - connecting to 127.0.0.1 11239
2017-01-24 22:13:10,156 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11239:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:35057
2017-01-24 22:13:10,156 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11239:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:35057
2017-01-24 22:13:10,156 [myid:1] - INFO [Thread-182:NIOServerCnxn$StatCommand@655] - Stat command output
2017-01-24 22:13:10,157 [myid:1] - INFO [Thread-182:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:35057 (no session established for client)
2017-01-24 22:13:10,157 [myid:] - INFO [Thread-170:ZooKeeper@438] - Initiating client connection, connectString=127.0.0.1:11236,127.0.0.1:11239 sessionTimeout=30000 watcher=org.apache.zookeeper.test.ClientBase$CountdownWatcher@6538b0f9
2017-01-24 22:13:10,158 [myid:] - WARN [Thread-170-SendThread(localhost:11236):ClientCnxn$SendThread@957] - SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: '/data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test2999887027410032136.junit.dir/jaas.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.
2017-01-24 22:13:10,158 [myid:] - INFO [Thread-170-SendThread(localhost:11236):ClientCnxn$SendThread@975] - Opening socket connection to server localhost/127.0.0.1:11236
2017-01-24 22:13:10,159 [myid:0] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11236:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:40338
2017-01-24 22:13:10,159 [myid:] - INFO [Thread-170-SendThread(localhost:11236):ClientCnxn$SendThread@852] - Socket connection established, initiating session, client: /127.0.0.1:40338, server: localhost/127.0.0.1:11236
2017-01-24 22:13:10,159 [myid:0] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11236:ZooKeeperServer@839] - Client attempting to establish new session at /127.0.0.1:40338
2017-01-24 22:13:10,160 [myid:1] - INFO [SyncThread:1:FileTxnLog@199] - Creating new log file: log.100000001
2017-01-24 22:13:10,160 [myid:0] - WARN [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11236:Follower@119] - Got zxid 0x100000001 expected 0x1
2017-01-24 22:13:10,160 [myid:0] - INFO [SyncThread:0:FileTxnLog@199] - Creating new log file: log.100000001
2017-01-24 22:13:10,163 [myid:0] - INFO [CommitProcessor:0:ZooKeeperServer@595] - Established session 0x59d441a96d0000 with negotiated timeout 30000 for client /127.0.0.1:40338
2017-01-24 22:13:10,163 [myid:] - INFO [Thread-170-SendThread(localhost:11236):ClientCnxn$SendThread@1235] - Session establishment complete on server localhost/127.0.0.1:11236, sessionid = 0x59d441a96d0000, negotiated timeout = 30000
2017-01-24 22:13:10,167 [myid:1] - INFO [ProcessThread(sid:1 cport:-1)::PrepRequestProcessor@494] - Processed session termination for sessionid: 0x59d441a96d0000
2017-01-24 22:13:10,168 [myid:0] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11236:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:40338 which had sessionid 0x59d441a96d0000
2017-01-24 22:13:10,169 [myid:] - INFO [Thread-170:ZooKeeper@684] - Session: 0x59d441a96d0000 closed
2017-01-24 22:13:10,169 [myid:] - INFO [Thread-170-EventThread:ClientCnxn$EventThread@512] - EventThread shut down
2017-01-24 22:13:10,169 [myid:] - INFO [Thread-170:JUnit4ZKTestRunner$LoggedInvokeMethod@55] - Memory used 27198
2017-01-24 22:13:10,169 [myid:] - INFO [Thread-170:JUnit4ZKTestRunner$LoggedInvokeMethod@60] - Number of threads 53
2017-01-24 22:13:10,169 [myid:] - INFO [Thread-170:JUnit4ZKTestRunner$LoggedInvokeMethod@65] - FINISHED TEST METHOD testNullAuthLearnerServer
2017-01-24 22:13:10,169 [myid:] - INFO [main:QuorumBase@314] - Shutting down quorum peer QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11236
2017-01-24 22:13:10,169 [myid:] - INFO [main:Follower@167] - shutdown called
java.lang.Exception: shutdown Follower
at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:167)
at org.apache.zookeeper.server.quorum.QuorumPeer.shutdown(QuorumPeer.java:896)
at org.apache.zookeeper.test.QuorumBase.shutdown(QuorumBase.java:315)
at org.apache.zookeeper.server.quorum.QuorumPeerTestBase$TestQPMain.shutdown(QuorumPeerTestBase.java:59)
at org.apache.zookeeper.server.quorum.QuorumPeerTestBase$MainThread.shutdown(QuorumPeerTestBase.java:152)
at org.apache.zookeeper.server.quorum.auth.QuorumAuthTestBase.shutdown(QuorumAuthTestBase.java:138)
at org.apache.zookeeper.server.quorum.auth.QuorumAuthTestBase.shutdownAll(QuorumAuthTestBase.java:131)
at org.apache.zookeeper.server.quorum.auth.QuorumAuthUpgradeTest.tearDown(QuorumAuthUpgradeTest.java:68)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:37)
at org.junit.rules.TestWatchman$1.evaluate(TestWatchman.java:48)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:76)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
at org.junit.runners.ParentRunner.run(ParentRunner.java:236)
at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:39)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:535)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1182)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:1033)
2017-01-24 22:13:10,170 [myid:] - INFO [main:FollowerZooKeeperServer@139] - Shutting down
2017-01-24 22:13:10,170 [myid:] - INFO [main:ZooKeeperServer@419] - shutting down
2017-01-24 22:13:10,170 [myid:] - INFO [main:FollowerRequestProcessor@105] - Shutting down
2017-01-24 22:13:10,170 [myid:] - INFO [main:CommitProcessor@181] - Shutting down
2017-01-24 22:13:10,170 [myid:] - INFO [main:FinalRequestProcessor@415] - shutdown of request processor complete
2017-01-24 22:13:10,170 [myid:0] - INFO [FollowerRequestProcessor:0:FollowerRequestProcessor@95] - FollowerRequestProcessor exited loop!
2017-01-24 22:13:10,170 [myid:0] - INFO [CommitProcessor:0:CommitProcessor@150] - CommitProcessor exited loop!
2017-01-24 22:13:10,171 [myid:] - INFO [main:SyncRequestProcessor@175] - Shutting down
2017-01-24 22:13:10,171 [myid:0] - INFO [SyncThread:0:SyncRequestProcessor@155] - SyncRequestProcessor exited!
2017-01-24 22:13:10,172 [myid:0] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11236:NIOServerCnxnFactory@224] - NIOServerCnxn factory exited run method
2017-01-24 22:13:10,172 [myid:0] - ERROR [localhost/127.0.0.1:11238:QuorumCnxManager$Listener@715] - Exception while listening
java.net.SocketException: Socket closed
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
at java.net.ServerSocket.implAccept(ServerSocket.java:530)
at java.net.ServerSocket.accept(ServerSocket.java:498)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$Listener.run(QuorumCnxManager.java:696)
2017-01-24 22:13:10,173 [myid:0] - WARN [SendWorker:1:QuorumCnxManager$SendWorker@896] - Interrupted while waiting for message on queue
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1049)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:73)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:884)
2017-01-24 22:13:10,173 [myid:1] - WARN [RecvWorker:0:QuorumCnxManager$RecvWorker@980] - Connection broken for id 0, my id = 1, error =
java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:392)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:965)
2017-01-24 22:13:10,173 [myid:1] - WARN [RecvWorker:0:QuorumCnxManager$RecvWorker@983] - Interrupting SendWorker
2017-01-24 22:13:10,173 [myid:0] - WARN [RecvWorker:1:QuorumCnxManager$RecvWorker@980] - Connection broken for id 1, my id = 0, error =
java.net.SocketException: Socket closed
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at java.io.BufferedInputStream.read(BufferedInputStream.java:254)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:965)
2017-01-24 22:13:10,174 [myid:0] - WARN [RecvWorker:1:QuorumCnxManager$RecvWorker@983] - Interrupting SendWorker
2017-01-24 22:13:10,174 [myid:1] - WARN [SendWorker:0:QuorumCnxManager$SendWorker@896] - Interrupted while waiting for message on queue
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1049)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:73)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:884)
2017-01-24 22:13:10,174 [myid:0] - WARN [SendWorker:1:QuorumCnxManager$SendWorker@906] - Send worker leaving thread
2017-01-24 22:13:10,175 [myid:1] - WARN [SendWorker:0:QuorumCnxManager$SendWorker@906] - Send worker leaving thread
2017-01-24 22:13:10,175 [myid:] - INFO [main:QuorumBase@318] - Shutting down leader election QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11236
2017-01-24 22:13:10,175 [myid:] - INFO [main:QuorumBase@323] - Waiting for QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11236 to exit thread
2017-01-24 22:13:10,230 [myid:] - WARN [Thread-12-SendThread(localhost:11230):ClientCnxn$SendThread@957] - SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: '/data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test2999887027410032136.junit.dir/jaas.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.
2017-01-24 22:13:10,231 [myid:] - INFO [Thread-12-SendThread(localhost:11230):ClientCnxn$SendThread@975] - Opening socket connection to server localhost/127.0.0.1:11230
2017-01-24 22:13:10,231 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11230:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:52552
2017-01-24 22:13:10,231 [myid:] - INFO [Thread-12-SendThread(localhost:11230):ClientCnxn$SendThread@852] - Socket connection established, initiating session, client: /127.0.0.1:52552, server: localhost/127.0.0.1:11230
2017-01-24 22:13:10,231 [myid:1] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11230:NIOServerCnxn@354] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2017-01-24 22:13:10,231 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11230:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:52552 (no session established for client)
2017-01-24 22:13:10,232 [myid:] - INFO [Thread-12-SendThread(localhost:11230):ClientCnxn$SendThread@1098] - Unable to read additional data from server sessionid 0x159d440f0ed0000, likely server has closed socket, closing socket connection and attempting reconnect
2017-01-24 22:13:10,277 [myid:1] - WARN [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11230:QuorumCnxManager@559] - Cannot open channel to 0 at election address localhost/127.0.0.1:11229
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:538)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:579)
at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:769)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:827)
2017-01-24 22:13:10,278 [myid:1] - WARN [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11230:QuorumCnxManager@559] - Cannot open channel to 2 at election address localhost/127.0.0.1:11235
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:538)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:579)
at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:769)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:827)
2017-01-24 22:13:10,278 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11230:FastLeaderElection@778] - Notification time out: 1600
2017-01-24 22:13:10,350 [myid:] - WARN [Thread-12-SendThread(localhost:11233):ClientCnxn$SendThread@957] - SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: '/data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test2999887027410032136.junit.dir/jaas.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.
2017-01-24 22:13:10,350 [myid:] - INFO [Thread-12-SendThread(localhost:11233):ClientCnxn$SendThread@975] - Opening socket connection to server localhost/127.0.0.1:11233
2017-01-24 22:13:10,351 [myid:] - WARN [Thread-12-SendThread(localhost:11233):ClientCnxn$SendThread@1102] - Session 0x159d440f0ed0000 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
2017-01-24 22:13:10,685 [myid:] - WARN [Thread-12-SendThread(localhost:11227):ClientCnxn$SendThread@957] - SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: '/data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test2999887027410032136.junit.dir/jaas.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.
2017-01-24 22:13:10,685 [myid:] - INFO [Thread-12-SendThread(localhost:11227):ClientCnxn$SendThread@975] - Opening socket connection to server localhost/127.0.0.1:11227
2017-01-24 22:13:10,685 [myid:] - WARN [Thread-12-SendThread(localhost:11227):ClientCnxn$SendThread@1102] - Session 0x159d440f0ed0000 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
2017-01-24 22:13:11,173 [myid:0] - INFO [localhost/127.0.0.1:11238:QuorumCnxManager$Listener@728] - Leaving listener
2017-01-24 22:13:11,873 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@340] - WorkerReceiver is down
2017-01-24 22:13:11,875 [myid:] - WARN [Thread-12-SendThread(localhost:11230):ClientCnxn$SendThread@957] - SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: '/data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test2999887027410032136.junit.dir/jaas.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.
2017-01-24 22:13:11,875 [myid:] - INFO [Thread-12-SendThread(localhost:11230):ClientCnxn$SendThread@975] - Opening socket connection to server localhost/127.0.0.1:11230
2017-01-24 22:13:11,875 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11230:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:52557
2017-01-24 22:13:11,876 [myid:] - INFO [Thread-12-SendThread(localhost:11230):ClientCnxn$SendThread@852] - Socket connection established, initiating session, client: /127.0.0.1:52557, server: localhost/127.0.0.1:11230
2017-01-24 22:13:11,876 [myid:1] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11230:NIOServerCnxn@354] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2017-01-24 22:13:11,876 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11230:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:52557 (no session established for client)
2017-01-24 22:13:11,876 [myid:] - INFO [Thread-12-SendThread(localhost:11230):ClientCnxn$SendThread@1098] - Unable to read additional data from server sessionid 0x159d440f0ed0000, likely server has closed socket, closing socket connection and attempting reconnect
2017-01-24 22:13:11,879 [myid:1] - WARN [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11230:QuorumCnxManager@559] - Cannot open channel to 0 at election address localhost/127.0.0.1:11229
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:538)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:579)
at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:769)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:827)
2017-01-24 22:13:11,879 [myid:1] - WARN [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11230:QuorumCnxManager@559] - Cannot open channel to 2 at election address localhost/127.0.0.1:11235
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:538)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:579)
at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:769)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:827)
2017-01-24 22:13:11,879 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11230:FastLeaderElection@778] - Notification time out: 3200
2017-01-24 22:13:11,918 [myid:0] - INFO [WorkerSender[myid=0]:FastLeaderElection$Messenger$WorkerSender@370] - WorkerSender is down
2017-01-24 22:13:11,919 [myid:0] - INFO [WorkerReceiver[myid=0]:FastLeaderElection$Messenger$WorkerReceiver@340] - WorkerReceiver is down
2017-01-24 22:13:12,132 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11236:Follower@167] - shutdown called
java.lang.Exception: shutdown Follower
at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:167)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:855)
2017-01-24 22:13:12,132 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11236:FollowerZooKeeperServer@139] - Shutting down
2017-01-24 22:13:12,132 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11236:ZooKeeperServer@419] - shutting down
2017-01-24 22:13:12,133 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11236:FollowerRequestProcessor@105] - Shutting down
2017-01-24 22:13:12,133 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11236:CommitProcessor@181] - Shutting down
2017-01-24 22:13:12,133 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11236:FinalRequestProcessor@415] - shutdown of request processor complete
2017-01-24 22:13:12,133 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11236:SyncRequestProcessor@175] - Shutting down
2017-01-24 22:13:12,133 [myid:0] - WARN [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11236:QuorumPeer@879] - QuorumPeer main thread exited
2017-01-24 22:13:12,135 [myid:] - INFO [main:ZKTestCase$1@60] - SUCCEEDED testNullAuthLearnerServer
2017-01-24 22:13:12,135 [myid:] - INFO [main:ZKTestCase$1@55] - FINISHED testNullAuthLearnerServer
2017-01-24 22:13:12,136 [myid:] - INFO [main:ZKTestCase$1@50] - STARTING testAuthLearnerAgainstNullAuthServer
2017-01-24 22:13:12,136 [myid:] - INFO [Thread-183:JUnit4ZKTestRunner$LoggedInvokeMethod@50] - RUNNING TEST METHOD testAuthLearnerAgainstNullAuthServer
2017-01-24 22:13:12,136 [myid:] - INFO [Thread-183:PortAssignment@32] - assigning port 11242
2017-01-24 22:13:12,136 [myid:] - INFO [Thread-183:PortAssignment@32] - assigning port 11243
2017-01-24 22:13:12,136 [myid:] - INFO [Thread-183:PortAssignment@32] - assigning port 11244
2017-01-24 22:13:12,137 [myid:] - INFO [Thread-183:PortAssignment@32] - assigning port 11245
2017-01-24 22:13:12,137 [myid:] - INFO [Thread-183:PortAssignment@32] - assigning port 11246
2017-01-24 22:13:12,137 [myid:] - INFO [Thread-183:PortAssignment@32] - assigning port 11247
2017-01-24 22:13:12,137 [myid:] - INFO [Thread-183:QuorumPeerTestBase$MainThread@81] - id = 0 tmpDir = /data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test7361117950796815905.junit.dir clientPort = 11242
2017-01-24 22:13:12,138 [myid:] - INFO [Thread-184:QuorumPeerConfig@111] - Reading configuration from: /data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test7361117950796815905.junit.dir/zoo.cfg
2017-01-24 22:13:12,138 [myid:] - INFO [Thread-183:QuorumPeerTestBase$MainThread@81] - id = 1 tmpDir = /data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test296771033193329447.junit.dir clientPort = 11245
2017-01-24 22:13:12,138 [myid:] - WARN [Thread-184:QuorumPeerConfig@327] - No server failure will be tolerated. You need at least 3 servers.
2017-01-24 22:13:12,138 [myid:] - INFO [Thread-184:QuorumPeerConfig@374] - Defaulting to majority quorums
2017-01-24 22:13:12,138 [myid:] - INFO [Thread-183:FourLetterWordMain@43] - connecting to 127.0.0.1 11242
2017-01-24 22:13:12,138 [myid:0] - INFO [Thread-184:DatadirCleanupManager@78] - autopurge.snapRetainCount set to 3
2017-01-24 22:13:12,139 [myid:0] - INFO [Thread-184:DatadirCleanupManager@79] - autopurge.purgeInterval set to 0
2017-01-24 22:13:12,139 [myid:0] - INFO [Thread-184:DatadirCleanupManager@101] - Purge task is not scheduled.
2017-01-24 22:13:12,139 [myid:] - INFO [Thread-183:ClientBase@246] - server 127.0.0.1:11242 not up java.net.ConnectException: Connection refused
2017-01-24 22:13:12,139 [myid:0] - WARN [Thread-184:QuorumPeerMain@129] - Unable to register log4j JMX control
javax.management.InstanceAlreadyExistsException: log4j:hiearchy=default
at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:437)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1898)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:966)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324)
at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
at org.apache.zookeeper.jmx.ManagedUtil.registerLog4jMBeans(ManagedUtil.java:53)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:127)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:116)
at org.apache.zookeeper.server.quorum.QuorumPeerTestBase$MainThread.run(QuorumPeerTestBase.java:140)
at java.lang.Thread.run(Thread.java:745)
2017-01-24 22:13:12,139 [myid:0] - INFO [Thread-184:QuorumPeerMain@132] - Starting quorum peer
2017-01-24 22:13:12,139 [myid:] - INFO [Thread-185:QuorumPeerConfig@111] - Reading configuration from: /data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test296771033193329447.junit.dir/zoo.cfg
2017-01-24 22:13:12,140 [myid:0] - INFO [Thread-184:NIOServerCnxnFactory@94] - binding to port 0.0.0.0/0.0.0.0:11242
2017-01-24 22:13:12,140 [myid:] - WARN [Thread-185:QuorumPeerConfig@327] - No server failure will be tolerated. You need at least 3 servers.
2017-01-24 22:13:12,140 [myid:] - INFO [Thread-185:QuorumPeerConfig@374] - Defaulting to majority quorums
2017-01-24 22:13:12,140 [myid:0] - INFO [Thread-184:QuorumPeer@1048] - minSessionTimeout set to -1
2017-01-24 22:13:12,140 [myid:1] - INFO [Thread-185:DatadirCleanupManager@78] - autopurge.snapRetainCount set to 3
2017-01-24 22:13:12,140 [myid:1] - INFO [Thread-185:DatadirCleanupManager@79] - autopurge.purgeInterval set to 0
2017-01-24 22:13:12,141 [myid:1] - INFO [Thread-185:DatadirCleanupManager@101] - Purge task is not scheduled.
2017-01-24 22:13:12,140 [myid:0] - INFO [Thread-184:QuorumPeer@1059] - maxSessionTimeout set to -1
2017-01-24 22:13:12,141 [myid:1] - WARN [Thread-185:QuorumPeerMain@129] - Unable to register log4j JMX control
javax.management.InstanceAlreadyExistsException: log4j:hiearchy=default
at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:437)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1898)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:966)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324)
at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
at org.apache.zookeeper.jmx.ManagedUtil.registerLog4jMBeans(ManagedUtil.java:53)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:127)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:116)
at org.apache.zookeeper.server.quorum.QuorumPeerTestBase$MainThread.run(QuorumPeerTestBase.java:140)
at java.lang.Thread.run(Thread.java:745)
2017-01-24 22:13:12,141 [myid:1] - INFO [Thread-185:QuorumPeerMain@132] - Starting quorum peer
2017-01-24 22:13:12,141 [myid:0] - INFO [Thread-184:QuorumPeer@1279] - quorum.auth.enableSasl set to true
2017-01-24 22:13:12,141 [myid:0] - INFO [Thread-184:QuorumPeer@1264] - quorum.auth.serverRequireSasl set to false
2017-01-24 22:13:12,141 [myid:0] - INFO [Thread-184:QuorumPeer@1270] - quorum.auth.learnerRequireSasl set to false
2017-01-24 22:13:12,141 [myid:1] - INFO [Thread-185:NIOServerCnxnFactory@94] - binding to port 0.0.0.0/0.0.0.0:11245
2017-01-24 22:13:12,142 [myid:0] - INFO [Thread-184:QuorumPeer@1286] - quorum.auth.kerberos.servicePrincipal set to zkquorum/localhost
2017-01-24 22:13:12,142 [myid:0] - INFO [Thread-184:QuorumPeer@1298] - quorum.auth.server.saslLoginContext set to QuorumServer
2017-01-24 22:13:12,142 [myid:0] - INFO [Thread-184:QuorumPeer@1292] - quorum.auth.learner.saslLoginContext set to QuorumLearner
2017-01-24 22:13:12,142 [myid:1] - INFO [Thread-185:QuorumPeer@1048] - minSessionTimeout set to -1
2017-01-24 22:13:12,142 [myid:1] - INFO [Thread-185:QuorumPeer@1059] - maxSessionTimeout set to -1
2017-01-24 22:13:12,142 [myid:0] - INFO [Thread-184:QuorumPeer@1306] - quorum.cnxn.threads.size set to 20
2017-01-24 22:13:12,142 [myid:1] - INFO [Thread-185:QuorumPeer@1277] - QuorumPeer communication is not secured!
2017-01-24 22:13:12,142 [myid:1] - INFO [Thread-185:QuorumPeer@1306] - quorum.cnxn.threads.size set to 20
2017-01-24 22:13:12,143 [myid:1] - INFO [Thread-185:QuorumPeer@540] - currentEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
2017-01-24 22:13:12,144 [myid:1] - INFO [Thread-185:QuorumPeer@555] - acceptedEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
2017-01-24 22:13:12,146 [myid:1] - INFO [Thread-186:QuorumCnxManager$Listener@691] - My election bind port: 0.0.0.0/0.0.0.0:11247
2017-01-24 22:13:12,146 [myid:0] - INFO [Thread-184:Login@294] - QuorumServer successfully logged in.
2017-01-24 22:13:12,146 [myid:0] - INFO [Thread-184:Login@294] - QuorumLearner successfully logged in.
2017-01-24 22:13:12,147 [myid:0] - INFO [Thread-184:QuorumPeer@540] - currentEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
2017-01-24 22:13:12,148 [myid:0] - INFO [Thread-184:QuorumPeer@555] - acceptedEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
2017-01-24 22:13:12,151 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11245:QuorumPeer@781] - LOOKING
2017-01-24 22:13:12,152 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11245:FastLeaderElection@744] - New election. My id = 1, proposed zxid=0x0
2017-01-24 22:13:12,152 [myid:1] - WARN [WorkerSender[myid=1]:QuorumCnxManager@559] - Cannot open channel to 0 at election address localhost/127.0.0.1:11244
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:538)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:514)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:393)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:365)
at java.lang.Thread.run(Thread.java:745)
2017-01-24 22:13:12,152 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@542] - Notification: 1 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)
2017-01-24 22:13:12,153 [myid:0] - INFO [Thread-187:QuorumCnxManager$Listener@691] - My election bind port: 0.0.0.0/0.0.0.0:11244
2017-01-24 22:13:12,158 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11242:QuorumPeer@781] - LOOKING
2017-01-24 22:13:12,158 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11242:FastLeaderElection@744] - New election. My id = 0, proposed zxid=0x0
2017-01-24 22:13:12,158 [myid:0] - INFO [WorkerReceiver[myid=0]:FastLeaderElection@542] - Notification: 0 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 0 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)
2017-01-24 22:13:12,159 [myid:1] - INFO [localhost/127.0.0.1:11247:QuorumCnxManager$Listener@698] - Received connection request /127.0.0.1:38550
2017-01-24 22:13:12,159 [myid:0] - INFO [QuorumConnectionThread-[myid=0]-1:SaslQuorumAuthLearner@79] - Skipping SASL authentication as quorum.auth.learnerRequireSasl=false
2017-01-24 22:13:12,160 [myid:0] - INFO [localhost/127.0.0.1:11244:QuorumCnxManager$Listener@698] - Received connection request /127.0.0.1:53591
2017-01-24 22:13:12,160 [myid:0] - INFO [QuorumConnectionThread-[myid=0]-1:QuorumCnxManager@331] - Have smaller server identifier, so dropping the connection: (1, 0)
2017-01-24 22:13:12,167 [myid:0] - INFO [WorkerReceiver[myid=0]:FastLeaderElection@542] - Notification: 1 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)
2017-01-24 22:13:12,167 [myid:0] - INFO [WorkerReceiver[myid=0]:FastLeaderElection@542] - Notification: 1 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 0 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)
2017-01-24 22:13:12,167 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@542] - Notification: 0 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 0 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)
2017-01-24 22:13:12,168 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@542] - Notification: 1 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 0 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)
2017-01-24 22:13:12,368 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11242:QuorumPeer@849] - FOLLOWING
2017-01-24 22:13:12,368 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11245:QuorumPeer@861] - LEADING
2017-01-24 22:13:12,368 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11242:ZooKeeperServer@162] - Created server with tickTime 4000 minSessionTimeout 8000 maxSessionTimeout 80000 datadir /data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test7361117950796815905.junit.dir/data/version-2 snapdir /data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test7361117950796815905.junit.dir/data/version-2
2017-01-24 22:13:12,368 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11245:ZooKeeperServer@162] - Created server with tickTime 4000 minSessionTimeout 8000 maxSessionTimeout 80000 datadir /data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test296771033193329447.junit.dir/data/version-2 snapdir /data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test296771033193329447.junit.dir/data/version-2
2017-01-24 22:13:12,368 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11242:Follower@64] - FOLLOWING - LEADER ELECTION TOOK - 210
2017-01-24 22:13:12,368 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11245:Leader@356] - LEADING - LEADER ELECTION TOOK - 216
2017-01-24 22:13:12,369 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11242:SaslQuorumAuthLearner@79] - Skipping SASL authentication as quorum.auth.learnerRequireSasl=false
2017-01-24 22:13:12,369 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11245:FileTxnSnapLog@281] - Snapshotting: 0x0 to /data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test296771033193329447.junit.dir/data/version-2/snapshot.0
2017-01-24 22:13:12,371 [myid:1] - INFO [LearnerHandler-/127.0.0.1:56735:LearnerHandler@287] - Follower sid: 0 : info : org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@68eefca4
2017-01-24 22:13:12,373 [myid:1] - INFO [LearnerHandler-/127.0.0.1:56735:LearnerHandler@342] - Synchronizing with Follower sid: 0 maxCommittedLog=0x0 minCommittedLog=0x0 peerLastZxid=0x0
2017-01-24 22:13:12,373 [myid:1] - INFO [LearnerHandler-/127.0.0.1:56735:LearnerHandler@441] - Sending snapshot last zxid of peer is 0x0 zxid of leader is 0x100000000sent zxid of db as 0x0
2017-01-24 22:13:12,373 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11242:Learner@329] - Getting a snapshot from leader
2017-01-24 22:13:12,374 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11242:FileTxnSnapLog@281] - Snapshotting: 0x0 to /data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test7361117950796815905.junit.dir/data/version-2/snapshot.0
2017-01-24 22:13:12,375 [myid:1] - INFO [LearnerHandler-/127.0.0.1:56735:LearnerHandler@477] - Received NEWLEADER-ACK message from 0
2017-01-24 22:13:12,375 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11245:Leader@934] - Have quorum of supporters, sids: [ 0,1 ]; starting up and setting last processed zxid: 0x100000000
2017-01-24 22:13:12,389 [myid:] - INFO [Thread-183:FourLetterWordMain@43] - connecting to 127.0.0.1 11242
2017-01-24 22:13:12,389 [myid:0] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11242:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:43163
2017-01-24 22:13:12,390 [myid:0] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11242:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:43163
2017-01-24 22:13:12,390 [myid:0] - INFO [Thread-190:NIOServerCnxn$StatCommand@655] - Stat command output
2017-01-24 22:13:12,391 [myid:0] - INFO [Thread-190:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:43163 (no session established for client)
2017-01-24 22:13:12,391 [myid:] - INFO [Thread-183:FourLetterWordMain@43] - connecting to 127.0.0.1 11245
2017-01-24 22:13:12,391 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11245:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:34938
2017-01-24 22:13:12,392 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11245:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:34938
2017-01-24 22:13:12,392 [myid:1] - INFO [Thread-191:NIOServerCnxn$StatCommand@655] - Stat command output
2017-01-24 22:13:12,393 [myid:1] - INFO [Thread-191:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:34938 (no session established for client)
2017-01-24 22:13:12,393 [myid:] - INFO [Thread-183:ZooKeeper@438] - Initiating client connection, connectString=127.0.0.1:11242,127.0.0.1:11245 sessionTimeout=30000 watcher=org.apache.zookeeper.test.ClientBase$CountdownWatcher@494b584c
2017-01-24 22:13:12,394 [myid:] - WARN [Thread-183-SendThread(localhost:11245):ClientCnxn$SendThread@957] - SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: '/data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test2999887027410032136.junit.dir/jaas.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.
2017-01-24 22:13:12,394 [myid:] - INFO [Thread-183-SendThread(localhost:11245):ClientCnxn$SendThread@975] - Opening socket connection to server localhost/127.0.0.1:11245
2017-01-24 22:13:12,394 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11245:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:34939
2017-01-24 22:13:12,394 [myid:] - INFO [Thread-183-SendThread(localhost:11245):ClientCnxn$SendThread@852] - Socket connection established, initiating session, client: /127.0.0.1:34939, server: localhost/127.0.0.1:11245
2017-01-24 22:13:12,395 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11245:ZooKeeperServer@839] - Client attempting to establish new session at /127.0.0.1:34939
2017-01-24 22:13:12,395 [myid:1] - INFO [SyncThread:1:FileTxnLog@199] - Creating new log file: log.100000001
2017-01-24 22:13:12,396 [myid:0] - WARN [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11242:Follower@119] - Got zxid 0x100000001 expected 0x1
2017-01-24 22:13:12,396 [myid:0] - INFO [SyncThread:0:FileTxnLog@199] - Creating new log file: log.100000001
2017-01-24 22:13:12,398 [myid:1] - INFO [CommitProcessor:1:ZooKeeperServer@595] - Established session 0x159d441b2380000 with negotiated timeout 30000 for client /127.0.0.1:34939
2017-01-24 22:13:12,398 [myid:] - INFO [Thread-183-SendThread(localhost:11245):ClientCnxn$SendThread@1235] - Session establishment complete on server localhost/127.0.0.1:11245, sessionid = 0x159d441b2380000, negotiated timeout = 30000
2017-01-24 22:13:12,405 [myid:1] - INFO [ProcessThread(sid:1 cport:-1)::PrepRequestProcessor@494] - Processed session termination for sessionid: 0x159d441b2380000
2017-01-24 22:13:12,408 [myid:] - INFO [Thread-183:ZooKeeper@684] - Session: 0x159d441b2380000 closed
2017-01-24 22:13:12,408 [myid:] - INFO [Thread-183-EventThread:ClientCnxn$EventThread@512] - EventThread shut down
2017-01-24 22:13:12,408 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11245:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:34939 which had sessionid 0x159d441b2380000
2017-01-24 22:13:12,408 [myid:] - INFO [Thread-183:JUnit4ZKTestRunner$LoggedInvokeMethod@55] - Memory used 30080
2017-01-24 22:13:12,409 [myid:] - INFO [Thread-183:JUnit4ZKTestRunner$LoggedInvokeMethod@60] - Number of threads 67
2017-01-24 22:13:12,409 [myid:] - INFO [Thread-183:JUnit4ZKTestRunner$LoggedInvokeMethod@65] - FINISHED TEST METHOD testAuthLearnerAgainstNullAuthServer
2017-01-24 22:13:12,409 [myid:] - INFO [main:QuorumBase@314] - Shutting down quorum peer QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11242
2017-01-24 22:13:12,409 [myid:] - INFO [main:Follower@167] - shutdown called
java.lang.Exception: shutdown Follower
at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:167)
at org.apache.zookeeper.server.quorum.QuorumPeer.shutdown(QuorumPeer.java:896)
at org.apache.zookeeper.test.QuorumBase.shutdown(QuorumBase.java:315)
at org.apache.zookeeper.server.quorum.QuorumPeerTestBase$TestQPMain.shutdown(QuorumPeerTestBase.java:59)
at org.apache.zookeeper.server.quorum.QuorumPeerTestBase$MainThread.shutdown(QuorumPeerTestBase.java:152)
at org.apache.zookeeper.server.quorum.auth.QuorumAuthTestBase.shutdown(QuorumAuthTestBase.java:138)
at org.apache.zookeeper.server.quorum.auth.QuorumAuthTestBase.shutdownAll(QuorumAuthTestBase.java:131)
at org.apache.zookeeper.server.quorum.auth.QuorumAuthUpgradeTest.tearDown(QuorumAuthUpgradeTest.java:68)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:37)
at org.junit.rules.TestWatchman$1.evaluate(TestWatchman.java:48)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:76)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
at org.junit.runners.ParentRunner.run(ParentRunner.java:236)
at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:39)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:535)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1182)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:1033)
2017-01-24 22:13:12,410 [myid:] - INFO [main:FollowerZooKeeperServer@139] - Shutting down
2017-01-24 22:13:12,410 [myid:] - INFO [main:ZooKeeperServer@419] - shutting down
2017-01-24 22:13:12,410 [myid:] - INFO [main:FollowerRequestProcessor@105] - Shutting down
2017-01-24 22:13:12,410 [myid:] - INFO [main:CommitProcessor@181] - Shutting down
2017-01-24 22:13:12,410 [myid:] - INFO [main:FinalRequestProcessor@415] - shutdown of request processor complete
2017-01-24 22:13:12,410 [myid:0] - INFO [FollowerRequestProcessor:0:FollowerRequestProcessor@95] - FollowerRequestProcessor exited loop!
2017-01-24 22:13:12,410 [myid:0] - INFO [CommitProcessor:0:CommitProcessor@150] - CommitProcessor exited loop!
2017-01-24 22:13:12,411 [myid:] - INFO [main:SyncRequestProcessor@175] - Shutting down
2017-01-24 22:13:12,411 [myid:0] - INFO [SyncThread:0:SyncRequestProcessor@155] - SyncRequestProcessor exited!
2017-01-24 22:13:12,411 [myid:0] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11242:NIOServerCnxnFactory@224] - NIOServerCnxn factory exited run method
2017-01-24 22:13:12,412 [myid:0] - ERROR [localhost/127.0.0.1:11244:QuorumCnxManager$Listener@715] - Exception while listening
java.net.SocketException: Socket closed
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
at java.net.ServerSocket.implAccept(ServerSocket.java:530)
at java.net.ServerSocket.accept(ServerSocket.java:498)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$Listener.run(QuorumCnxManager.java:696)
2017-01-24 22:13:12,412 [myid:1] - WARN [RecvWorker:0:QuorumCnxManager$RecvWorker@980] - Connection broken for id 0, my id = 1, error =
java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:392)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:965)
2017-01-24 22:13:12,412 [myid:0] - WARN [RecvWorker:1:QuorumCnxManager$RecvWorker@980] - Connection broken for id 1, my id = 0, error =
java.net.SocketException: Socket closed
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at java.io.BufferedInputStream.read(BufferedInputStream.java:254)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:965)
2017-01-24 22:13:12,413 [myid:0] - WARN [SendWorker:1:QuorumCnxManager$SendWorker@896] - Interrupted while waiting for message on queue
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1049)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:73)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:884)
2017-01-24 22:13:12,414 [myid:0] - WARN [SendWorker:1:QuorumCnxManager$SendWorker@906] - Send worker leaving thread
2017-01-24 22:13:12,412 [myid:1] - WARN [RecvWorker:0:QuorumCnxManager$RecvWorker@983] - Interrupting SendWorker
2017-01-24 22:13:12,414 [myid:] - INFO [main:QuorumBase@318] - Shutting down leader election QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11242
2017-01-24 22:13:12,413 [myid:0] - WARN [RecvWorker:1:QuorumCnxManager$RecvWorker@983] - Interrupting SendWorker
2017-01-24 22:13:12,415 [myid:] - INFO [main:QuorumBase@323] - Waiting for QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11242 to exit thread
2017-01-24 22:13:12,414 [myid:1] - WARN [SendWorker:0:QuorumCnxManager$SendWorker@896] - Interrupted while waiting for message on queue
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1049)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:73)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:884)
2017-01-24 22:13:12,415 [myid:1] - WARN [SendWorker:0:QuorumCnxManager$SendWorker@906] - Send worker leaving thread
2017-01-24 22:13:12,656 [myid:] - WARN [Thread-12-SendThread(localhost:11233):ClientCnxn$SendThread@957] - SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: '/data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test2999887027410032136.junit.dir/jaas.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.
2017-01-24 22:13:12,656 [myid:] - INFO [Thread-12-SendThread(localhost:11233):ClientCnxn$SendThread@975] - Opening socket connection to server localhost/127.0.0.1:11233
2017-01-24 22:13:12,656 [myid:] - WARN [Thread-12-SendThread(localhost:11233):ClientCnxn$SendThread@1102] - Session 0x159d440f0ed0000 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
2017-01-24 22:13:13,412 [myid:0] - INFO [localhost/127.0.0.1:11244:QuorumCnxManager$Listener@728] - Leaving listener
2017-01-24 22:13:13,603 [myid:] - WARN [Thread-12-SendThread(localhost:11227):ClientCnxn$SendThread@957] - SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: '/data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test2999887027410032136.junit.dir/jaas.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.
2017-01-24 22:13:13,603 [myid:] - INFO [Thread-12-SendThread(localhost:11227):ClientCnxn$SendThread@975] - Opening socket connection to server localhost/127.0.0.1:11227
2017-01-24 22:13:13,604 [myid:] - WARN [Thread-12-SendThread(localhost:11227):ClientCnxn$SendThread@1102] - Session 0x159d440f0ed0000 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
2017-01-24 22:13:14,382 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11242:Follower@167] - shutdown called
java.lang.Exception: shutdown Follower
at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:167)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:855)
2017-01-24 22:13:14,382 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11242:FollowerZooKeeperServer@139] - Shutting down
2017-01-24 22:13:14,382 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11242:ZooKeeperServer@419] - shutting down
2017-01-24 22:13:14,382 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11242:FollowerRequestProcessor@105] - Shutting down
2017-01-24 22:13:14,382 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11242:CommitProcessor@181] - Shutting down
2017-01-24 22:13:14,382 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11242:FinalRequestProcessor@415] - shutdown of request processor complete
2017-01-24 22:13:14,383 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11242:SyncRequestProcessor@175] - Shutting down
2017-01-24 22:13:14,383 [myid:0] - WARN [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11242:QuorumPeer@879] - QuorumPeer main thread exited
2017-01-24 22:13:14,384 [myid:] - INFO [main:ZKTestCase$1@60] - SUCCEEDED testAuthLearnerAgainstNullAuthServer
2017-01-24 22:13:14,384 [myid:] - INFO [main:ZKTestCase$1@55] - FINISHED testAuthLearnerAgainstNullAuthServer
2017-01-24 22:13:14,385 [myid:] - INFO [main:ZKTestCase$1@50] - STARTING testAuthLearnerServer
2017-01-24 22:13:14,385 [myid:] - INFO [Thread-192:JUnit4ZKTestRunner$LoggedInvokeMethod@50] - RUNNING TEST METHOD testAuthLearnerServer
2017-01-24 22:13:14,385 [myid:] - INFO [Thread-192:PortAssignment@32] - assigning port 11248
2017-01-24 22:13:14,386 [myid:] - INFO [Thread-192:PortAssignment@32] - assigning port 11249
2017-01-24 22:13:14,386 [myid:] - INFO [Thread-192:PortAssignment@32] - assigning port 11250
2017-01-24 22:13:14,386 [myid:] - INFO [Thread-192:PortAssignment@32] - assigning port 11251
2017-01-24 22:13:14,386 [myid:] - INFO [Thread-192:PortAssignment@32] - assigning port 11252
2017-01-24 22:13:14,386 [myid:] - INFO [Thread-192:PortAssignment@32] - assigning port 11253
2017-01-24 22:13:14,386 [myid:] - INFO [Thread-192:QuorumPeerTestBase$MainThread@81] - id = 0 tmpDir = /data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test4413362313903870178.junit.dir clientPort = 11248
2017-01-24 22:13:14,387 [myid:] - INFO [Thread-193:QuorumPeerConfig@111] - Reading configuration from: /data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test4413362313903870178.junit.dir/zoo.cfg
2017-01-24 22:13:14,387 [myid:] - INFO [Thread-192:QuorumPeerTestBase$MainThread@81] - id = 1 tmpDir = /data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test8693106920335238483.junit.dir clientPort = 11251
2017-01-24 22:13:14,387 [myid:] - WARN [Thread-193:QuorumPeerConfig@327] - No server failure will be tolerated. You need at least 3 servers.
2017-01-24 22:13:14,388 [myid:] - INFO [Thread-193:QuorumPeerConfig@374] - Defaulting to majority quorums
2017-01-24 22:13:14,388 [myid:] - INFO [Thread-192:FourLetterWordMain@43] - connecting to 127.0.0.1 11248
2017-01-24 22:13:14,388 [myid:0] - INFO [Thread-193:DatadirCleanupManager@78] - autopurge.snapRetainCount set to 3
2017-01-24 22:13:14,388 [myid:0] - INFO [Thread-193:DatadirCleanupManager@79] - autopurge.purgeInterval set to 0
2017-01-24 22:13:14,388 [myid:0] - INFO [Thread-193:DatadirCleanupManager@101] - Purge task is not scheduled.
2017-01-24 22:13:14,388 [myid:] - INFO [Thread-192:ClientBase@246] - server 127.0.0.1:11248 not up java.net.ConnectException: Connection refused
2017-01-24 22:13:14,388 [myid:0] - WARN [Thread-193:QuorumPeerMain@129] - Unable to register log4j JMX control
javax.management.InstanceAlreadyExistsException: log4j:hiearchy=default
at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:437)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1898)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:966)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324)
at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
at org.apache.zookeeper.jmx.ManagedUtil.registerLog4jMBeans(ManagedUtil.java:53)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:127)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:116)
at org.apache.zookeeper.server.quorum.QuorumPeerTestBase$MainThread.run(QuorumPeerTestBase.java:140)
at java.lang.Thread.run(Thread.java:745)
2017-01-24 22:13:14,389 [myid:0] - INFO [Thread-193:QuorumPeerMain@132] - Starting quorum peer
2017-01-24 22:13:14,388 [myid:] - INFO [Thread-194:QuorumPeerConfig@111] - Reading configuration from: /data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test8693106920335238483.junit.dir/zoo.cfg
2017-01-24 22:13:14,389 [myid:0] - INFO [Thread-193:NIOServerCnxnFactory@94] - binding to port 0.0.0.0/0.0.0.0:11248
2017-01-24 22:13:14,389 [myid:] - WARN [Thread-194:QuorumPeerConfig@327] - No server failure will be tolerated. You need at least 3 servers.
2017-01-24 22:13:14,389 [myid:] - INFO [Thread-194:QuorumPeerConfig@374] - Defaulting to majority quorums
2017-01-24 22:13:14,390 [myid:1] - INFO [Thread-194:DatadirCleanupManager@78] - autopurge.snapRetainCount set to 3
2017-01-24 22:13:14,390 [myid:1] - INFO [Thread-194:DatadirCleanupManager@79] - autopurge.purgeInterval set to 0
2017-01-24 22:13:14,390 [myid:1] - INFO [Thread-194:DatadirCleanupManager@101] - Purge task is not scheduled.
2017-01-24 22:13:14,390 [myid:0] - INFO [Thread-193:QuorumPeer@1048] - minSessionTimeout set to -1
2017-01-24 22:13:14,390 [myid:0] - INFO [Thread-193:QuorumPeer@1059] - maxSessionTimeout set to -1
2017-01-24 22:13:14,390 [myid:1] - WARN [Thread-194:QuorumPeerMain@129] - Unable to register log4j JMX control
javax.management.InstanceAlreadyExistsException: log4j:hiearchy=default
at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:437)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1898)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:966)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324)
at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
at org.apache.zookeeper.jmx.ManagedUtil.registerLog4jMBeans(ManagedUtil.java:53)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:127)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:116)
at org.apache.zookeeper.server.quorum.QuorumPeerTestBase$MainThread.run(QuorumPeerTestBase.java:140)
at java.lang.Thread.run(Thread.java:745)
2017-01-24 22:13:14,391 [myid:1] - INFO [Thread-194:QuorumPeerMain@132] - Starting quorum peer
2017-01-24 22:13:14,391 [myid:0] - INFO [Thread-193:QuorumPeer@1279] - quorum.auth.enableSasl set to true
2017-01-24 22:13:14,391 [myid:0] - INFO [Thread-193:QuorumPeer@1264] - quorum.auth.serverRequireSasl set to true
2017-01-24 22:13:14,391 [myid:0] - INFO [Thread-193:QuorumPeer@1270] - quorum.auth.learnerRequireSasl set to true
2017-01-24 22:13:14,391 [myid:1] - INFO [Thread-194:NIOServerCnxnFactory@94] - binding to port 0.0.0.0/0.0.0.0:11251
2017-01-24 22:13:14,391 [myid:0] - INFO [Thread-193:QuorumPeer@1286] - quorum.auth.kerberos.servicePrincipal set to zkquorum/localhost
2017-01-24 22:13:14,391 [myid:0] - INFO [Thread-193:QuorumPeer@1298] - quorum.auth.server.saslLoginContext set to QuorumServer
2017-01-24 22:13:14,392 [myid:0] - INFO [Thread-193:QuorumPeer@1292] - quorum.auth.learner.saslLoginContext set to QuorumLearner
2017-01-24 22:13:14,392 [myid:1] - INFO [Thread-194:QuorumPeer@1048] - minSessionTimeout set to -1
2017-01-24 22:13:14,392 [myid:1] - INFO [Thread-194:QuorumPeer@1059] - maxSessionTimeout set to -1
2017-01-24 22:13:14,392 [myid:0] - INFO [Thread-193:QuorumPeer@1306] - quorum.cnxn.threads.size set to 20
2017-01-24 22:13:14,392 [myid:1] - INFO [Thread-194:QuorumPeer@1279] - quorum.auth.enableSasl set to true
2017-01-24 22:13:14,392 [myid:1] - INFO [Thread-194:QuorumPeer@1264] - quorum.auth.serverRequireSasl set to true
2017-01-24 22:13:14,392 [myid:1] - INFO [Thread-194:QuorumPeer@1270] - quorum.auth.learnerRequireSasl set to true
2017-01-24 22:13:14,392 [myid:1] - INFO [Thread-194:QuorumPeer@1286] - quorum.auth.kerberos.servicePrincipal set to zkquorum/localhost
2017-01-24 22:13:14,392 [myid:1] - INFO [Thread-194:QuorumPeer@1298] - quorum.auth.server.saslLoginContext set to QuorumServer
2017-01-24 22:13:14,393 [myid:1] - INFO [Thread-194:QuorumPeer@1292] - quorum.auth.learner.saslLoginContext set to QuorumLearner
2017-01-24 22:13:14,393 [myid:1] - INFO [Thread-194:QuorumPeer@1306] - quorum.cnxn.threads.size set to 20
2017-01-24 22:13:14,393 [myid:0] - INFO [Thread-193:Login@294] - QuorumServer successfully logged in.
2017-01-24 22:13:14,397 [myid:1] - INFO [Thread-194:Login@294] - QuorumServer successfully logged in.
2017-01-24 22:13:14,397 [myid:0] - INFO [Thread-193:Login@294] - QuorumLearner successfully logged in.
2017-01-24 22:13:14,397 [myid:1] - INFO [Thread-194:Login@294] - QuorumLearner successfully logged in.
2017-01-24 22:13:14,397 [myid:0] - INFO [Thread-193:QuorumPeer@540] - currentEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
2017-01-24 22:13:14,398 [myid:1] - INFO [Thread-194:QuorumPeer@540] - currentEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
2017-01-24 22:13:14,398 [myid:0] - INFO [Thread-193:QuorumPeer@555] - acceptedEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
2017-01-24 22:13:14,399 [myid:1] - INFO [Thread-194:QuorumPeer@555] - acceptedEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
2017-01-24 22:13:14,405 [myid:0] - INFO [Thread-195:QuorumCnxManager$Listener@691] - My election bind port: 0.0.0.0/0.0.0.0:11250
2017-01-24 22:13:14,405 [myid:1] - INFO [Thread-196:QuorumCnxManager$Listener@691] - My election bind port: 0.0.0.0/0.0.0.0:11253
2017-01-24 22:13:14,406 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11251:QuorumPeer@781] - LOOKING
2017-01-24 22:13:14,406 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11248:QuorumPeer@781] - LOOKING
2017-01-24 22:13:14,407 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11251:FastLeaderElection@744] - New election. My id = 1, proposed zxid=0x0
2017-01-24 22:13:14,407 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11248:FastLeaderElection@744] - New election. My id = 0, proposed zxid=0x0
2017-01-24 22:13:14,407 [myid:0] - INFO [WorkerReceiver[myid=0]:FastLeaderElection@542] - Notification: 0 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 0 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)
2017-01-24 22:13:14,407 [myid:0] - INFO [localhost/127.0.0.1:11250:QuorumCnxManager$Listener@698] - Received connection request /127.0.0.1:60553
2017-01-24 22:13:14,408 [myid:1] - INFO [localhost/127.0.0.1:11253:QuorumCnxManager$Listener@698] - Received connection request /127.0.0.1:59854
2017-01-24 22:13:14,408 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@542] - Notification: 1 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)
2017-01-24 22:13:14,411 [myid:1] - INFO [QuorumConnectionThread-[myid=1]-1:SecurityUtils@68] - QuorumLearner will use DIGEST-MD5 as SASL mechanism.
2017-01-24 22:13:14,412 [myid:0] - INFO [QuorumConnectionThread-[myid=0]-1:SecurityUtils@68] - QuorumLearner will use DIGEST-MD5 as SASL mechanism.
2017-01-24 22:13:14,415 [myid:0] - INFO [QuorumConnectionThread-[myid=0]-2:SaslQuorumServerCallbackHandler@143] - Successfully authenticated learner: authenticationID=test; authorizationID=test.
2017-01-24 22:13:14,415 [myid:1] - INFO [QuorumConnectionThread-[myid=1]-2:SaslQuorumServerCallbackHandler@143] - Successfully authenticated learner: authenticationID=test; authorizationID=test.
2017-01-24 22:13:14,416 [myid:1] - INFO [QuorumConnectionThread-[myid=1]-2:SaslQuorumAuthServer@114] - Successfully completed the authentication using SASL. learner addr: /127.0.0.1:59854
2017-01-24 22:13:14,416 [myid:1] - INFO [QuorumConnectionThread-[myid=1]-1:SaslQuorumAuthLearner@151] - Successfully completed the authentication using SASL. server addr: localhost/127.0.0.1:11250, status: SUCCESS
2017-01-24 22:13:14,416 [myid:0] - INFO [QuorumConnectionThread-[myid=0]-2:SaslQuorumAuthServer@114] - Successfully completed the authentication using SASL. learner addr: /127.0.0.1:60553
2017-01-24 22:13:14,416 [myid:0] - INFO [QuorumConnectionThread-[myid=0]-1:SaslQuorumAuthLearner@151] - Successfully completed the authentication using SASL. server addr: localhost/127.0.0.1:11253, status: SUCCESS
2017-01-24 22:13:14,417 [myid:0] - INFO [QuorumConnectionThread-[myid=0]-1:QuorumCnxManager@331] - Have smaller server identifier, so dropping the connection: (1, 0)
2017-01-24 22:13:14,427 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@542] - Notification: 0 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 0 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)
2017-01-24 22:13:14,428 [myid:0] - INFO [WorkerReceiver[myid=0]:FastLeaderElection@542] - Notification: 1 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)
2017-01-24 22:13:14,428 [myid:0] - INFO [WorkerReceiver[myid=0]:FastLeaderElection@542] - Notification: 1 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 0 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)
2017-01-24 22:13:14,428 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@542] - Notification: 1 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 0 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)
2017-01-24 22:13:14,628 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11248:QuorumPeer@849] - FOLLOWING
2017-01-24 22:13:14,629 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11248:ZooKeeperServer@162] - Created server with tickTime 4000 minSessionTimeout 8000 maxSessionTimeout 80000 datadir /data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test4413362313903870178.junit.dir/data/version-2 snapdir /data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test4413362313903870178.junit.dir/data/version-2
2017-01-24 22:13:14,629 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11251:QuorumPeer@861] - LEADING
2017-01-24 22:13:14,629 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11248:Follower@64] - FOLLOWING - LEADER ELECTION TOOK - 222
2017-01-24 22:13:14,629 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11251:ZooKeeperServer@162] - Created server with tickTime 4000 minSessionTimeout 8000 maxSessionTimeout 80000 datadir /data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test8693106920335238483.junit.dir/data/version-2 snapdir /data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test8693106920335238483.junit.dir/data/version-2
2017-01-24 22:13:14,629 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11251:Leader@356] - LEADING - LEADER ELECTION TOOK - 222
2017-01-24 22:13:14,630 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11248:SecurityUtils@68] - QuorumLearner will use DIGEST-MD5 as SASL mechanism.
2017-01-24 22:13:14,630 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11251:FileTxnSnapLog@281] - Snapshotting: 0x0 to /data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test8693106920335238483.junit.dir/data/version-2/snapshot.0
2017-01-24 22:13:14,632 [myid:1] - INFO [Thread-197:SaslQuorumServerCallbackHandler@143] - Successfully authenticated learner: authenticationID=test; authorizationID=test.
2017-01-24 22:13:14,633 [myid:1] - INFO [Thread-197:SaslQuorumAuthServer@114] - Successfully completed the authentication using SASL. learner addr: /127.0.0.1:36656
2017-01-24 22:13:14,634 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11248:SaslQuorumAuthLearner@151] - Successfully completed the authentication using SASL. server addr: localhost/127.0.0.1:11252, status: SUCCESS
2017-01-24 22:13:14,634 [myid:1] - INFO [LearnerHandler-/127.0.0.1:36656:LearnerHandler@287] - Follower sid: 0 : info : org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@5b8ec669
2017-01-24 22:13:14,636 [myid:1] - INFO [LearnerHandler-/127.0.0.1:36656:LearnerHandler@342] - Synchronizing with Follower sid: 0 maxCommittedLog=0x0 minCommittedLog=0x0 peerLastZxid=0x0
2017-01-24 22:13:14,636 [myid:1] - INFO [LearnerHandler-/127.0.0.1:36656:LearnerHandler@441] - Sending snapshot last zxid of peer is 0x0 zxid of leader is 0x100000000sent zxid of db as 0x0
2017-01-24 22:13:14,636 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11248:Learner@329] - Getting a snapshot from leader
2017-01-24 22:13:14,638 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11248:FileTxnSnapLog@281] - Snapshotting: 0x0 to /data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test4413362313903870178.junit.dir/data/version-2/snapshot.0
2017-01-24 22:13:14,638 [myid:] - INFO [Thread-192:FourLetterWordMain@43] - connecting to 127.0.0.1 11248
2017-01-24 22:13:14,639 [myid:0] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11248:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:52527
2017-01-24 22:13:14,639 [myid:1] - INFO [LearnerHandler-/127.0.0.1:36656:LearnerHandler@477] - Received NEWLEADER-ACK message from 0
2017-01-24 22:13:14,639 [myid:0] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11248:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:52527
2017-01-24 22:13:14,643 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11251:Leader@934] - Have quorum of supporters, sids: [ 0,1 ]; starting up and setting last processed zxid: 0x100000000
2017-01-24 22:13:14,647 [myid:0] - INFO [Thread-199:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:52527 (no session established for client)
2017-01-24 22:13:14,898 [myid:] - INFO [Thread-192:FourLetterWordMain@43] - connecting to 127.0.0.1 11248
2017-01-24 22:13:14,898 [myid:0] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11248:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:52528
2017-01-24 22:13:14,899 [myid:0] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11248:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:52528
2017-01-24 22:13:14,899 [myid:0] - INFO [Thread-200:NIOServerCnxn$StatCommand@655] - Stat command output
2017-01-24 22:13:14,900 [myid:0] - INFO [Thread-200:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:52528 (no session established for client)
2017-01-24 22:13:14,900 [myid:] - INFO [Thread-192:FourLetterWordMain@43] - connecting to 127.0.0.1 11251
2017-01-24 22:13:14,900 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11251:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:59828
2017-01-24 22:13:14,901 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11251:NIOServerCnxn@821] - Processing stat command from /127.0.0.1:59828
2017-01-24 22:13:14,901 [myid:1] - INFO [Thread-201:NIOServerCnxn$StatCommand@655] - Stat command output
2017-01-24 22:13:14,902 [myid:1] - INFO [Thread-201:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:59828 (no session established for client)
2017-01-24 22:13:14,902 [myid:] - INFO [Thread-192:ZooKeeper@438] - Initiating client connection, connectString=127.0.0.1:11248,127.0.0.1:11251 sessionTimeout=30000 watcher=org.apache.zookeeper.test.ClientBase$CountdownWatcher@76e543fa
2017-01-24 22:13:14,903 [myid:] - WARN [Thread-192-SendThread(localhost:11251):ClientCnxn$SendThread@957] - SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: '/data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test2999887027410032136.junit.dir/jaas.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.
2017-01-24 22:13:14,903 [myid:] - INFO [Thread-192-SendThread(localhost:11251):ClientCnxn$SendThread@975] - Opening socket connection to server localhost/127.0.0.1:11251
2017-01-24 22:13:14,903 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11251:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:59829
2017-01-24 22:13:14,903 [myid:] - INFO [Thread-192-SendThread(localhost:11251):ClientCnxn$SendThread@852] - Socket connection established, initiating session, client: /127.0.0.1:59829, server: localhost/127.0.0.1:11251
2017-01-24 22:13:14,904 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11251:ZooKeeperServer@839] - Client attempting to establish new session at /127.0.0.1:59829
2017-01-24 22:13:14,904 [myid:1] - INFO [SyncThread:1:FileTxnLog@199] - Creating new log file: log.100000001
2017-01-24 22:13:14,904 [myid:0] - WARN [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11248:Follower@119] - Got zxid 0x100000001 expected 0x1
2017-01-24 22:13:14,905 [myid:0] - INFO [SyncThread:0:FileTxnLog@199] - Creating new log file: log.100000001
2017-01-24 22:13:14,907 [myid:1] - INFO [CommitProcessor:1:ZooKeeperServer@595] - Established session 0x159d441bb130000 with negotiated timeout 30000 for client /127.0.0.1:59829
2017-01-24 22:13:14,907 [myid:] - INFO [Thread-192-SendThread(localhost:11251):ClientCnxn$SendThread@1235] - Session establishment complete on server localhost/127.0.0.1:11251, sessionid = 0x159d441bb130000, negotiated timeout = 30000
2017-01-24 22:13:14,910 [myid:1] - INFO [ProcessThread(sid:1 cport:-1)::PrepRequestProcessor@494] - Processed session termination for sessionid: 0x159d441bb130000
2017-01-24 22:13:14,912 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11251:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:59829 which had sessionid 0x159d441bb130000
2017-01-24 22:13:14,912 [myid:] - INFO [Thread-192:ZooKeeper@684] - Session: 0x159d441bb130000 closed
2017-01-24 22:13:14,912 [myid:] - INFO [Thread-192:JUnit4ZKTestRunner$LoggedInvokeMethod@55] - Memory used 33263
2017-01-24 22:13:14,912 [myid:] - INFO [Thread-192:JUnit4ZKTestRunner$LoggedInvokeMethod@60] - Number of threads 85
2017-01-24 22:13:14,912 [myid:] - INFO [Thread-192:JUnit4ZKTestRunner$LoggedInvokeMethod@65] - FINISHED TEST METHOD testAuthLearnerServer
2017-01-24 22:13:14,913 [myid:] - INFO [main:QuorumBase@314] - Shutting down quorum peer QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11248
2017-01-24 22:13:14,913 [myid:] - INFO [Thread-192-EventThread:ClientCnxn$EventThread@512] - EventThread shut down
2017-01-24 22:13:14,913 [myid:] - INFO [main:Follower@167] - shutdown called
java.lang.Exception: shutdown Follower
at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:167)
at org.apache.zookeeper.server.quorum.QuorumPeer.shutdown(QuorumPeer.java:896)
at org.apache.zookeeper.test.QuorumBase.shutdown(QuorumBase.java:315)
at org.apache.zookeeper.server.quorum.QuorumPeerTestBase$TestQPMain.shutdown(QuorumPeerTestBase.java:59)
at org.apache.zookeeper.server.quorum.QuorumPeerTestBase$MainThread.shutdown(QuorumPeerTestBase.java:152)
at org.apache.zookeeper.server.quorum.auth.QuorumAuthTestBase.shutdown(QuorumAuthTestBase.java:138)
at org.apache.zookeeper.server.quorum.auth.QuorumAuthTestBase.shutdownAll(QuorumAuthTestBase.java:131)
at org.apache.zookeeper.server.quorum.auth.QuorumAuthUpgradeTest.tearDown(QuorumAuthUpgradeTest.java:68)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:37)
at org.junit.rules.TestWatchman$1.evaluate(TestWatchman.java:48)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:76)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
at org.junit.runners.ParentRunner.run(ParentRunner.java:236)
at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:39)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:535)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1182)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:1033)
2017-01-24 22:13:14,913 [myid:] - INFO [main:FollowerZooKeeperServer@139] - Shutting down
2017-01-24 22:13:14,913 [myid:] - INFO [main:ZooKeeperServer@419] - shutting down
2017-01-24 22:13:14,914 [myid:] - INFO [main:FollowerRequestProcessor@105] - Shutting down
2017-01-24 22:13:14,914 [myid:] - INFO [main:CommitProcessor@181] - Shutting down
2017-01-24 22:13:14,914 [myid:] - INFO [main:FinalRequestProcessor@415] - shutdown of request processor complete
2017-01-24 22:13:14,914 [myid:0] - INFO [FollowerRequestProcessor:0:FollowerRequestProcessor@95] - FollowerRequestProcessor exited loop!
2017-01-24 22:13:14,914 [myid:0] - INFO [CommitProcessor:0:CommitProcessor@150] - CommitProcessor exited loop!
2017-01-24 22:13:14,914 [myid:] - INFO [main:SyncRequestProcessor@175] - Shutting down
2017-01-24 22:13:14,915 [myid:0] - INFO [SyncThread:0:SyncRequestProcessor@155] - SyncRequestProcessor exited!
2017-01-24 22:13:14,915 [myid:0] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11248:NIOServerCnxnFactory@224] - NIOServerCnxn factory exited run method
2017-01-24 22:13:14,916 [myid:0] - ERROR [localhost/127.0.0.1:11250:QuorumCnxManager$Listener@715] - Exception while listening
java.net.SocketException: Socket closed
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
at java.net.ServerSocket.implAccept(ServerSocket.java:530)
at java.net.ServerSocket.accept(ServerSocket.java:498)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$Listener.run(QuorumCnxManager.java:696)
2017-01-24 22:13:14,917 [myid:1] - WARN [RecvWorker:0:QuorumCnxManager$RecvWorker@980] - Connection broken for id 0, my id = 1, error =
java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:392)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:965)
2017-01-24 22:13:14,917 [myid:0] - WARN [RecvWorker:1:QuorumCnxManager$RecvWorker@980] - Connection broken for id 1, my id = 0, error =
java.net.SocketException: Socket closed
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at java.io.BufferedInputStream.read(BufferedInputStream.java:254)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:965)
2017-01-24 22:13:14,917 [myid:0] - WARN [RecvWorker:1:QuorumCnxManager$RecvWorker@983] - Interrupting SendWorker
2017-01-24 22:13:14,917 [myid:0] - WARN [SendWorker:1:QuorumCnxManager$SendWorker@896] - Interrupted while waiting for message on queue
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1049)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:73)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:884)
2017-01-24 22:13:14,917 [myid:1] - WARN [RecvWorker:0:QuorumCnxManager$RecvWorker@983] - Interrupting SendWorker
2017-01-24 22:13:14,918 [myid:1] - WARN [SendWorker:0:QuorumCnxManager$SendWorker@896] - Interrupted while waiting for message on queue
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1049)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:73)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:884)
2017-01-24 22:13:14,918 [myid:0] - WARN [SendWorker:1:QuorumCnxManager$SendWorker@906] - Send worker leaving thread
2017-01-24 22:13:14,919 [myid:1] - WARN [SendWorker:0:QuorumCnxManager$SendWorker@906] - Send worker leaving thread
2017-01-24 22:13:14,920 [myid:] - INFO [main:QuorumBase@318] - Shutting down leader election QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11248
2017-01-24 22:13:14,920 [myid:] - INFO [main:QuorumBase@323] - Waiting for QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11248 to exit thread
2017-01-24 22:13:15,080 [myid:1] - WARN [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11230:QuorumCnxManager@559] - Cannot open channel to 0 at election address localhost/127.0.0.1:11229
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:538)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:579)
at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:769)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:827)
2017-01-24 22:13:15,080 [myid:1] - WARN [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11230:QuorumCnxManager@559] - Cannot open channel to 2 at election address localhost/127.0.0.1:11235
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:538)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:579)
at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:769)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:827)
2017-01-24 22:13:15,081 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11230:FastLeaderElection@778] - Notification time out: 6400
2017-01-24 22:13:15,167 [myid:0] - INFO [WorkerSender[myid=0]:FastLeaderElection$Messenger$WorkerSender@370] - WorkerSender is down
2017-01-24 22:13:15,167 [myid:0] - INFO [WorkerReceiver[myid=0]:FastLeaderElection$Messenger$WorkerReceiver@340] - WorkerReceiver is down
2017-01-24 22:13:15,664 [myid:] - WARN [Thread-12-SendThread(localhost:11230):ClientCnxn$SendThread@957] - SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: '/data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test2999887027410032136.junit.dir/jaas.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.
2017-01-24 22:13:15,664 [myid:] - INFO [Thread-12-SendThread(localhost:11230):ClientCnxn$SendThread@975] - Opening socket connection to server localhost/127.0.0.1:11230
2017-01-24 22:13:15,664 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11230:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:52580
2017-01-24 22:13:15,664 [myid:] - INFO [Thread-12-SendThread(localhost:11230):ClientCnxn$SendThread@852] - Socket connection established, initiating session, client: /127.0.0.1:52580, server: localhost/127.0.0.1:11230
2017-01-24 22:13:15,665 [myid:1] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11230:NIOServerCnxn@354] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2017-01-24 22:13:15,665 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11230:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:52580 (no session established for client)
2017-01-24 22:13:15,665 [myid:] - INFO [Thread-12-SendThread(localhost:11230):ClientCnxn$SendThread@1098] - Unable to read additional data from server sessionid 0x159d440f0ed0000, likely server has closed socket, closing socket connection and attempting reconnect
2017-01-24 22:13:15,917 [myid:0] - INFO [localhost/127.0.0.1:11250:QuorumCnxManager$Listener@728] - Leaving listener
2017-01-24 22:13:16,602 [myid:] - WARN [Thread-12-SendThread(localhost:11233):ClientCnxn$SendThread@957] - SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: '/data/jenkins/workspace/CDH5-ZooKeeper-3.4.5-JDK7/build/test/tmp/test2999887027410032136.junit.dir/jaas.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.
2017-01-24 22:13:16,602 [myid:] - INFO [Thread-12-SendThread(localhost:11233):ClientCnxn$SendThread@975] - Opening socket connection to server localhost/127.0.0.1:11233
2017-01-24 22:13:16,602 [myid:] - WARN [Thread-12-SendThread(localhost:11233):ClientCnxn$SendThread@1102] - Session 0x159d440f0ed0000 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
2017-01-24 22:13:16,651 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11248:Follower@167] - shutdown called
java.lang.Exception: shutdown Follower
at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:167)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:855)
2017-01-24 22:13:16,651 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11248:FollowerZooKeeperServer@139] - Shutting down
2017-01-24 22:13:16,651 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11248:ZooKeeperServer@419] - shutting down
2017-01-24 22:13:16,651 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11248:FollowerRequestProcessor@105] - Shutting down
2017-01-24 22:13:16,651 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11248:CommitProcessor@181] - Shutting down
2017-01-24 22:13:16,652 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11248:FinalRequestProcessor@415] - shutdown of request processor complete
2017-01-24 22:13:16,652 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11248:SyncRequestProcessor@175] - Shutting down
2017-01-24 22:13:16,652 [myid:0] - WARN [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11248:QuorumPeer@879] - QuorumPeer main thread exited
2017-01-24 22:13:16,653 [myid:] - INFO [main:ZKTestCase$1@60] - SUCCEEDED testAuthLearnerServer
2017-01-24 22:13:16,654 [myid:] - INFO [main:ZKTestCase$1@55] - FINISHED testAuthLearnerServer

{noformat}

flaky, flaky-test 9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
1 year, 21 weeks ago 0|i399sn:
ZooKeeper ZOOKEEPER-2678

Large databases take a long time to regain a quorum

Bug Closed Major Fixed Robert Joseph Evans Robert Joseph Evans Robert Joseph Evans 26/Jan/17 10:16   15/Jul/17 01:13 14/Feb/17 13:06 3.4.9, 3.5.2 3.4.10, 3.5.3, 3.6.0 server   1 12   I know this is long but please here me out.

I recently inherited a massive zookeeper ensemble. The snapshot is 3.4 GB on disk. Because of its massive size we have been running into a number of issues. There are lots of problems that we hope to fix with tuning GC etc, but the big one right now that is blocking us making a lot of progress on the rest of them is that when we lose a quorum because the leader left, for what ever reason, it can take well over 5 mins for a new quorum to be established. So we cannot tune the leader without risking downtime.

We traced down where the time was being spent and found that each server was clearing the database so it would be read back in again before leader election even started. Then as part of the sync phase each server will write out a snapshot to checkpoint the progress it made as part of the sync.

I will be putting up a patch shortly with some proposed changes in it.
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
2 years, 48 weeks, 2 days ago 0|i3990v:
ZooKeeper ZOOKEEPER-2677

Verify the occurrence of CancelledKeyException in zookeeper branch-3.5 and above

Task Open Major Unresolved Michael Han Rakesh Radhakrishnan Rakesh Radhakrishnan 26/Jan/17 01:20   31/Jan/17 11:41   3.5.3, 3.6.0       0 1   Red Hat Enterprise Linux Server release 6.2 As per the [discussion|https://issues.apache.org/jira/browse/ZOOKEEPER-2044?focusedCommentId=15836893&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15836893] in ZOOKEEPER-2044 jira, it need to analyse the chance of {{CancelledKeyException}} and fix(if any) in branch-3.5 and master. This jira task can be used for this. 9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 8 weeks ago 0|i3989j:
ZooKeeper ZOOKEEPER-2676

Enable remote debugging unit tests on CLI

Improvement Resolved Minor Won't Fix Unassigned Edward Ribeiro Edward Ribeiro 25/Jan/17 07:20   06/Nov/19 13:52 06/Nov/19 13:52         1 1   Sometimes it's useful to be able to run a unit test from the CLI and then attach an IDE to enable debugging as below:

{code}
$ ant -Dtestcase=FourLetterWordsTest -DremoteDebug=true test-core-java
{code}

The unit test will stop as below:

{code}
(...)
junit.run-concurrent:
[echo] Running 1 concurrent JUnit processes.
[junit] Listening for transport dt_socket at address: 5005
{code}

And we will be able to put breakpoints on the target class and bind the IDE to it's process to step through the test.
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 8 weeks, 1 day ago 0|i396mf:
ZooKeeper ZOOKEEPER-2675

Bump Mockito to version 2

Improvement Open Trivial Unresolved Unassigned Edward Ribeiro Edward Ribeiro 25/Jan/17 06:06   09/Dec/19 08:07           0 2   Current mockito version is 1.8.2, but version 2 brings new improvements while keeping backwards compatibility to jdk6 (branch-3.4) and support partially jdk8. So, this issue is to bring mockito version up to date. 9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
14 weeks, 3 days ago 0|i396gn:
ZooKeeper ZOOKEEPER-2674

Projet-Final

Bug Resolved Trivial Invalid Unassigned berkani berkani 24/Jan/17 09:33   31/Jan/17 11:38 31/Jan/17 11:38         0 2   9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 7 weeks, 2 days ago 0|i394q7:
ZooKeeper ZOOKEEPER-2673

projet

Bug Resolved Trivial Invalid Unassigned berkani berkani 24/Jan/17 08:27   31/Jan/17 11:38 31/Jan/17 11:38         0 1   9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 7 weeks, 2 days ago 0|i394nr:
ZooKeeper ZOOKEEPER-2672

Remove CHANGE.txt

Improvement Closed Major Fixed Michael Han Michael Han Michael Han 23/Jan/17 13:26   31/Mar/17 05:01 09/Feb/17 22:08 3.4.9, 3.5.2 3.4.10, 3.5.3, 3.6.0 build   0 7   The CHANGE.txt is already not the source of truth of what's changed after we migrating to git - most of the git commits in recent couple of months don't update CHANGE.txt. The option of updating CHANGE.txt during commit flow automatically is none trivial, and do that manually is cumbersome and error prone.

The consensus is we would rely on source control revision logs instead of CHANGE.txt moving forward; see https://www.mail-archive.com/dev@zookeeper.apache.org/msg37108.html for more details.

9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 5 weeks, 3 days ago 0|i3937r:
ZooKeeper ZOOKEEPER-2671

Fix compilation error in branch-3.4

Bug Closed Major Fixed Rakesh Radhakrishnan Mohammad Arshad Mohammad Arshad 22/Jan/17 22:59   31/Mar/17 05:01 23/Jan/17 01:55   3.4.10 server   0 5   branch-3.4 code compilation is failing. Following are the compilation erros:
{code}
compile-test:
[mkdir] Created dir: D:\gitHome\zookeeperTrunk\build\test\classes
[javac] Compiling 146 source files to D:\gitHome\zookeeperTrunk\build\test\classes
[javac] warning: [options] bootstrap class path not set in conjunction with -source 1.6
[javac] D:\gitHome\zookeeperTrunk\src\java\test\org\apache\zookeeper\server\PurgeTxnTest.java:464: error: cannot find symbol
[javac] ZooKeeper zk = ClientBase.createZKClient(HOSTPORT);
[javac] ^
[javac] symbol: method createZKClient(String)
[javac] location: class ClientBase
[javac] D:\gitHome\zookeeperTrunk\src\java\test\org\apache\zookeeper\server\PurgeTxnTest.java:503: error: cannot find symbol
[javac] zk = ClientBase.createZKClient(HOSTPORT);
[javac] ^
[javac] symbol: method createZKClient(String)
[javac] location: class ClientBase
[javac] Note: Some input files use or override a deprecated API.
{code}
9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
3 years, 8 weeks, 3 days ago 0|i3924f:
ZooKeeper ZOOKEEPER-2670

CLONE - Connections fo ZooKeeper server becomes slow over time with native GSSAPI

Bug Resolved Major Duplicate Enis Soztutar Yan Fitterer Yan Fitterer 19/Jan/17 14:32   09/Feb/17 12:52 09/Feb/17 12:52 3.4.6, 3.4.7, 3.4.8, 3.5.0 3.4.6, 3.4.7, 3.4.8, 3.5.2 server   0 4   OS: RHEL6
Java: 1.8.0_40
Configuration:

java.env:
{noformat}
SERVER_JVMFLAGS="$SERVER_JVMFLAGS -Xmx5120m"
SERVER_JVMFLAGS="$SERVER_JVMFLAGS -Djava.security.auth.login.config=/local/apps/zookeeper-test1/conf/jaas-server.conf"
SERVER_JVMFLAGS="$SERVER_JVMFLAGS -Dsun.security.jgss.native=true"
{noformat}

jaas-server.conf:
{noformat}
Server {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
isInitiator=false
principal="zookeeper/<hostname>@<REALM>";
};
{noformat}

Process environment:
{noformat}
KRB5_KTNAME=/local/apps/zookeeper-test1/conf/keytab
ZOO_LOG_DIR=/local/apps/zookeeper-test1/log
ZOOCFGDIR=/local/apps/zookeeper-test1/conf
{noformat}
ZooKeeper server becomes slow over time when native GSSAPI is used. The connection to the server starts taking upto 10 seconds.
This is happening with ZooKeeper-3.4.6 and is fairly reproducible.

Debug logs:
{noformat}
2015-07-02 00:58:49,318 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:42405:NIOServerCnxnFactory@197] - Accepted socket connection from /<client_ip>:47942
2015-07-02 00:58:49,318 [myid:] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:42405:ZooKeeperSaslServer@78] - serviceHostname is '<zookeeper-server>'
2015-07-02 00:58:49,318 [myid:] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:42405:ZooKeeperSaslServer@79] - servicePrincipalName is 'zookeeper'
2015-07-02 00:58:49,318 [myid:] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:42405:ZooKeeperSaslServer@80] - SASL mechanism(mech) is 'GSSAPI'
2015-07-02 00:58:49,324 [myid:] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:42405:ZooKeeperSaslServer@106] - Added private credential to subject: [GSSCredential:
zookeeper@<zookeeper-server> 1.2.840.113554.1.2.2 Accept [class sun.security.jgss.wrapper.GSSCredElement]]
2015-07-02 00:58:59,441 [myid:] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:42405:ZooKeeperServer@810] - Session establishment request from client /<client_ip>:47942 client's lastZxid is 0x0
2015-07-02 00:58:59,441 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:42405:ZooKeeperServer@868] - Client attempting to establish new session at /<client_ip>:47942
2015-07-02 00:58:59,448 [myid:] - DEBUG [SyncThread:0:FinalRequestProcessor@88] - Processing request:: sessionid:0x14e486028785c81 type:createSession cxid:0x0 zxid:0x110e79 txntype:-10 reqpath:n/a
2015-07-02 00:58:59,448 [myid:] - DEBUG [SyncThread:0:FinalRequestProcessor@160] - sessionid:0x14e486028785c81 type:createSession cxid:0x0 zxid:0x110e79 txntype:-10 reqpath:n/a
2015-07-02 00:58:59,448 [myid:] - INFO [SyncThread:0:ZooKeeperServer@617] - Established session 0x14e486028785c81 with negotiated timeout 10000 for client /<client_ip>:47942
2015-07-02 00:58:59,452 [myid:] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:42405:ZooKeeperServer@949] - Responding to client SASL token.
2015-07-02 00:58:59,452 [myid:] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:42405:ZooKeeperServer@953] - Size of client SASL token: 706
2015-07-02 00:58:59,460 [myid:] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:42405:ZooKeeperServer@984] - Size of server SASL response: 161
2015-07-02 00:58:59,462 [myid:] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:42405:ZooKeeperServer@949] - Responding to client SASL token.
2015-07-02 00:58:59,462 [myid:] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:42405:ZooKeeperServer@953] - Size of client SASL token: 0
2015-07-02 00:58:59,462 [myid:] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:42405:ZooKeeperServer@984] - Size of server SASL response: 32
2015-07-02 00:58:59,463 [myid:] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:42405:ZooKeeperServer@949] - Responding to client SASL token.
2015-07-02 00:58:59,463 [myid:] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:42405:ZooKeeperServer@953] - Size of client SASL token: 32
2015-07-02 00:58:59,464 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:42405:SaslServerCallbackHandler@118] - Successfully authenticated client: authenticationID=<user_principal>; authorizationID=<user_principal>.
2015-07-02 00:58:59,464 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:42405:ZooKeeperServer@964] - adding SASL authorization for authorizationID: <user_principal>
2015-07-02 00:58:59,465 [myid:] - INFO [ProcessThread(sid:0 cport:-1)::PrepRequestProcessor@494] - Processed session termination for sessionid: 0x14e486028785c81
2015-07-02 00:58:59,467 [myid:] - DEBUG [SyncThread:0:FinalRequestProcessor@88] - Processing request:: sessionid:0x14e486028785c81 type:closeSession cxid:0x1 zxid:0x110e7a txntype:-11 reqpath:n/a
2015-07-02 00:58:59,467 [myid:] - DEBUG [SyncThread:0:FinalRequestProcessor@160] - sessionid:0x14e486028785c81 type:closeSession cxid:0x1 zxid:0x110e7a txntype:-11 reqpath:n/a
2015-07-02 00:58:59,467 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:42405:NIOServerCnxn@1007] - Closed socket connection for client /<client_ip>:47942 which had sessionid 0x14e486028785c81
{noformat}

If you see, after adding the credentials to privateCredential set, it takes roughly 10 seconds to reach to session establishment request. From the code it looks like Subject.doAs() is taking a lot of time.

I connected it to jdb while it was waiting and got following stacktrace:
{noformat}
NIOServerCxn.Factory:0.0.0.0/0.0.0.0:58909:
[1] java.util.HashMap$TreeNode.find (HashMap.java:1,865)
[2] java.util.HashMap$TreeNode.find (HashMap.java:1,861)
[3] java.util.HashMap$TreeNode.find (HashMap.java:1,861)
[4] java.util.HashMap$TreeNode.find (HashMap.java:1,861)
[5] java.util.HashMap$TreeNode.find (HashMap.java:1,861)
[6] java.util.HashMap$TreeNode.find (HashMap.java:1,861)
[7] java.util.HashMap$TreeNode.find (HashMap.java:1,861)
[8] java.util.HashMap$TreeNode.putTreeVal (HashMap.java:1,981)
[9] java.util.HashMap.putVal (HashMap.java:637)
[10] java.util.HashMap.put (HashMap.java:611)
[11] java.util.HashSet.add (HashSet.java:219)
[12] javax.security.auth.Subject$ClassSet.populateSet (Subject.java:1,418)
[13] javax.security.auth.Subject$ClassSet.<init> (Subject.java:1,372)
[14] javax.security.auth.Subject.getPrivateCredentials (Subject.java:767)
[15] sun.security.jgss.GSSUtil$1.run (GSSUtil.java:340)
[16] sun.security.jgss.GSSUtil$1.run (GSSUtil.java:332)
[17] java.security.AccessController.doPrivileged (native method)
[18] sun.security.jgss.GSSUtil.searchSubject (GSSUtil.java:332)
[19] sun.security.jgss.wrapper.NativeGSSFactory.getCredFromSubject (NativeGSSFactory.java:53)
[20] sun.security.jgss.wrapper.NativeGSSFactory.getCredentialElement (NativeGSSFactory.java:116)
[21] sun.security.jgss.GSSManagerImpl.getCredentialElement (GSSManagerImpl.java:193)
[22] sun.security.jgss.GSSCredentialImpl.add (GSSCredentialImpl.java:427)
[23] sun.security.jgss.GSSCredentialImpl.<init> (GSSCredentialImpl.java:62)
[24] sun.security.jgss.GSSManagerImpl.createCredential (GSSManagerImpl.java:154)
[25] com.sun.security.sasl.gsskerb.GssKrb5Server.<init> (GssKrb5Server.java:108)
[26] com.sun.security.sasl.gsskerb.FactoryImpl.createSaslServer (FactoryImpl.java:85)
[27] javax.security.sasl.Sasl.createSaslServer (Sasl.java:524)
[28] org.apache.zookeeper.server.ZooKeeperSaslServer$1.run (ZooKeeperSaslServer.java:118)
[29] org.apache.zookeeper.server.ZooKeeperSaslServer$1.run (ZooKeeperSaslServer.java:114)
[30] java.security.AccessController.doPrivileged (native method)
[31] javax.security.auth.Subject.doAs (Subject.java:422)
[32] org.apache.zookeeper.server.ZooKeeperSaslServer.createSaslServer (ZooKeeperSaslServer.java:114)
[33] org.apache.zookeeper.server.ZooKeeperSaslServer.<init> (ZooKeeperSaslServer.java:48)
[34] org.apache.zookeeper.server.NIOServerCnxn.<init> (NIOServerCnxn.java:100)
[35] org.apache.zookeeper.server.NIOServerCnxnFactory.createConnection (NIOServerCnxnFactory.java:161)
[36] org.apache.zookeeper.server.NIOServerCnxnFactory.run (NIOServerCnxnFactory.java:202)
[37] java.lang.Thread.run (Thread.java:745)
{noformat}

This doesn't happen when we use JGSS, I think because adding credential to privateCredential set for every connection is causing Subject.doAS() to take much longer time.
patch 9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 6 weeks ago Fix slowness in connections when setup with native GSSAPI. kerberos, native-gssapi 0|i38y6f:
ZooKeeper ZOOKEEPER-2669

follower failed to reconnect to leader after a network error

Bug Open Major Unresolved Unassigned Zhenghua Chen Zhenghua Chen 19/Jan/17 06:30   29/Jul/19 14:44   3.4.9   quorum, server   0 9   CentOS7 We have a zookeeper cluster with 3 nodes named s1, s2, s3
By mistake, we shut down the ethernet interface of s2, and zk follower shut down(zk process remains there)
Later, after ethernet up again, s2 failed to reconnect to leader s3 to be a follower

follower s2 keeps printing log like this:
{quote}
2017-01-19 16:40:58,956 WARN [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:7181] o.a.z.s.q.Learner - Got zxid 0x320001019f expected 0x1
2017-01-19 16:40:58,956 ERROR [SyncThread:1] o.a.z.s.ZooKeeperCriticalThread - Severe unrecoverable error, from thread : SyncThread:1
java.nio.channels.ClosedChannelException: null
at sun.nio.ch.FileChannelImpl.ensureOpen(FileChannelImpl.java:99)
at sun.nio.ch.FileChannelImpl.position(FileChannelImpl.java:250)
at org.apache.zookeeper.server.persistence.Util.padLogFile(Util.java:215)
at org.apache.zookeeper.server.persistence.FileTxnLog.padFile(FileTxnLog.java:241)
at org.apache.zookeeper.server.persistence.FileTxnLog.append(FileTxnLog.java:219)
at org.apache.zookeeper.server.persistence.FileTxnSnapLog.append(FileTxnSnapLog.java:314)
at org.apache.zookeeper.server.ZKDatabase.append(ZKDatabase.java:470)
at org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:140)
2017-01-19 16:40:58,956 INFO [SyncThread:1] o.a.z.s.ZooKeeperServerListenerImpl - Thread SyncThread:1 exits, error code 1
2017-01-19 16:40:58,956 INFO [SyncThread:1] o.a.z.s.SyncRequestProcessor - SyncRequestProcessor exited!
2017-01-19 16:40:58,957 INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:7181] o.a.z.s.q.Learner - shutdown called
java.lang.Exception: shutdown Follower
at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:164)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:850)
{quote}

And, leader s3 keeps printing log like this:
{quote}
2017-01-19 16:30:50,452 INFO [LearnerHandler-/192.168.40.51:35949] o.a.z.s.q.LearnerHandler - Follower sid: 1 : info : org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@95258f0
2017-01-19 16:30:50,452 INFO [LearnerHandler-/192.168.40.51:35949] o.a.z.s.q.LearnerHandler - Synchronizing with Follower sid: 1 maxCommittedLog=0x320001019e minCommittedLog=0x320000ffaa peerLastZxid=0x2300000000
2017-01-19 16:30:50,453 WARN [LearnerHandler-/192.168.40.51:35949] o.a.z.s.q.LearnerHandler - Unhandled proposal scenario
2017-01-19 16:30:50,453 INFO [LearnerHandler-/192.168.40.51:35949] o.a.z.s.q.LearnerHandler - Sending SNAP
2017-01-19 16:30:50,453 INFO [LearnerHandler-/192.168.40.51:35949] o.a.z.s.q.LearnerHandler - Sending snapshot last zxid of peer is 0x2300000000 zxid of leader is 0x320001019esent zxid of db as 0x320001019e
2017-01-19 16:30:50,461 INFO [LearnerHandler-/192.168.40.51:35949] o.a.z.s.q.LearnerHandler - Received NEWLEADER-ACK message from 1
2017-01-19 16:30:51,738 ERROR [LearnerHandler-/192.168.40.51:35934] o.a.z.s.q.LearnerHandler - Unexpected exception causing shutdown while sock still open
java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at java.io.BufferedInputStream.read(BufferedInputStream.java:254)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63)
at org.apache.zookeeper.server.quorum.QuorumPacket.deserialize(QuorumPacket.java:83)
at org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:99)
at org.apache.zookeeper.server.quorum.LearnerHandler.run(LearnerHandler.java:542)
{quote}

we execute netstat, found lots of close wait socket in s2, and never closed.
{quote}
tcp6 10865 0 192.168.40.51:47181 192.168.40.57:7288 CLOSE_WAIT 2217/java
tcp6 2576 0 192.168.40.51:57181 192.168.40.57:7288 CLOSE_WAIT 2217/java
{quote}
seems that s2 has a connection leak.

after restart zk process of s2, it works fine.
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
33 weeks, 3 days ago 0|i38xdz:
ZooKeeper ZOOKEEPER-2668

Remove reference to requireClientAuthScheme from https://cwiki.apache.org/confluence/display/ZOOKEEPER/Client-Server+mutual+authentication

Bug Open Major Unresolved Unassigned Devaraj Das Devaraj Das 18/Jan/17 17:55   17/Sep/19 11:28       documentation   0 6   I was trying to see if ZK can be configured to always do client authentication (globally and not per znode). I came across this https://cwiki.apache.org/confluence/display/ZOOKEEPER/Client-Server+mutual+authentication and it describes a config key requireClientAuthScheme that when set to 'sasl' should do the job. But upon looking at code (in master and in branch-3.5), I don't see any reference to it.
Raising this jira to update the wiki (assuming I am on the right track). There are probably ways to update the wiki otherwise but I wanted to get some attention on this before we did that. (cc [~phunt], [~ekoontz]).
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
26 weeks, 2 days ago 0|i38wf3:
ZooKeeper ZOOKEEPER-2667

NPE in the patch for ZOOKEEPER-2139 when multiple connections are made

Bug Resolved Major Invalid Unassigned Hari Krishna Dara Hari Krishna Dara 17/Jan/17 06:35   18/Jan/17 07:23 18/Jan/17 07:23 3.5.2, 3.6.0   java client   0 3   ZOOKEEPER-2139 added support for connecting to multiple ZK services, but this also introduced a bug that causes a cryptic NPE. The client sees the below sort of error messages:

{noformat}
Exception while trying to create SASL client: java.lang.NullPointerException
SASL authentication with Zookeeper Quorum member failed: javax.security.sasl.SaslException: saslClient failed to initialize properly: it's null.
Error while calling watcher
java.lang.NullPointerException
at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:581)
at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:532)
at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:40)
at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:579)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:554)
{noformat}

The line at {{ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:581)}} points to the middle line below, where {{event.getState()}} is {{null}}:

{noformat}
private void connectionEvent(WatchedEvent event) {
switch(event.getState()) {
case SyncConnected:
{noformat}

However, the event's state is {{null}} because of a couple of other bugs, particularly an NPE that gets a mention in the log without a stacktrace. This first NPE causes an incorrect initialization of the event and results in the second NPE with the stacktrace.

The reason for the first NPE comes from this code in {{ZookeeperSaslClient}}:

{noformat}
if (!initializedLogin) {
...
}
Subject subject = login.getSubject();
{noformat}

Before the patch for ZOOKEEPER-2139, both the {{login}} and {{initializedLogin}} were {{static}} fields of {{ZookeeperSaslClient}}. To support multiple ZK clients, the {{login}} field was changed from {{static}} to instance field, however the {{initializedLogin}} field was left as {{static}} field. Because of this, the subsequent attempts to connect to ZK think that the login doesn't need to be done and go ahead and blindly use the {{login}} variable which causes the NPE.

At the core, the fix is simply to change {{initializedLogin}} to instance variable, but we have made a few additional changes to improve the logging and handle state. I will attach a patch soon.
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 9 weeks, 1 day ago 0|i38t5j:
ZooKeeper ZOOKEEPER-2666

the watch function called many times when it should be called once

Bug Open Major Unresolved Unassigned ZHE CHEN ZHE CHEN 16/Jan/17 00:51   23/Jan/17 17:38   3.4.5   c client, server   0 3   ubuntu 14.04
zookeeper 3.4.5, it's installed by apt-get
We have a service A, which has 2 instances A1 and A2.
We also have another 2 services, B and C. B has 2 instances B1 and B2. C has 2 instances C1 and C2.

A1 and A2 both register child watch for B and C. 2 individual watches, of course.

I restart B1 and C1 nearly at the same time. Then, theoretically A1 and A2
both should receive 2 events about the child change of service B and C.
However, the real result is, A1 received the 2 children changes of service B and C separately, A2 only received the children change of service B. Moreover, A2 got the children change of service B many many times when service B only changed once at that time (I add auto re-registration so A2 can receive the event more than once).

Till now, it only happened once. If it happens again, maybe I will provide some logs.
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 8 weeks, 3 days ago 0|i38qsn:
ZooKeeper ZOOKEEPER-2665

Port QA github pull request build to branch 3.4 and 3.5

Test Closed Major Fixed Enrico Olivelli Flavio Paiva Junqueira Flavio Paiva Junqueira 14/Jan/17 12:34   18/May/17 00:00 07/Mar/17 19:33   3.4.10, 3.5.3 build   0 6   We have QA build for pull requests against master but not against branches 3.4 and 3.5. We need to port the necessary wiring to do it, it shouldn't be difficult. 9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
2 years, 44 weeks ago 0|i38pun:
ZooKeeper ZOOKEEPER-2664

ClientPortBindTest#testBindByAddress may fail due to "No such device" exception

Test Closed Major Fixed Ted Yu Ted Yu Ted Yu 13/Jan/17 05:04   31/Mar/17 05:01 14/Jan/17 00:32 3.4.6 3.4.10, 3.5.3, 3.6.0     0 4   Saw the following in a recent run:
{code}
Stacktrace

java.net.SocketException: No such device
at java.net.NetworkInterface.isLoopback0(Native Method)
at java.net.NetworkInterface.isLoopback(NetworkInterface.java:390)
at org.apache.zookeeper.test.ClientPortBindTest.testBindByAddress(ClientPortBindTest.java:61)
at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:52)
Standard Output

2017-01-12 23:20:43,792 [myid:] - INFO [main:ZKTestCase$1@50] - STARTING testBindByAddress
2017-01-12 23:20:43,795 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@50] - RUNNING TEST METHOD testBindByAddress
2017-01-12 23:20:43,799 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@62] - TEST METHOD FAILED testBindByAddress
java.net.SocketException: No such device
at java.net.NetworkInterface.isLoopback0(Native Method)
at java.net.NetworkInterface.isLoopback(NetworkInterface.java:390)
at org.apache.zookeeper.test.ClientPortBindTest.testBindByAddress(ClientPortBindTest.java:61)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:52)
at org.junit.rules.TestWatchman$1.evaluate(TestWatchman.java:48)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:76)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184)
at org.junit.runners.ParentRunner.run(ParentRunner.java:236)
at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:39)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:532)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1179)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:1030)
{code}
Proposed fix is to catch exception from isLoopback() call.
9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
3 years, 8 weeks, 3 days ago 0|i38ngn:
ZooKeeper ZOOKEEPER-2663

Enable remote jmx, zkCli.sh start failed with jmx communication error

Bug Open Major Unresolved Unassigned linbo.liao linbo.liao 11/Jan/17 08:41   11/Jan/17 08:41   3.4.9   jmx   0 2   OS: Centos 6.7 x86_64
Zookeeper: 3.4.9
Java: HotSpot 1.8.0_65-b17
My laptop is Macbook Pro with macOS Sierra (IP: 192.168.2.102). An VM (IP: 192.168.2.107) is running on VirtualBox.

Deploy zookeeper-3.4.9 on VM, enable the remote JMX with option:

-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=8415 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.rmi.port=8415
-Djava.rmi.server.hostname=192.168.2.107

Test with jconsole on Mac, connect 192.168.2.107:8415 works fine.

Runnign zkCli.sh failed
$ bin/zkCli.sh
Error: JMX connector server communication error: service:jmx:rmi://localhost.localdomain:8415

$ cat /etc/hosts
127.0.0.1 localhost.localdomain localhost
::1 localhost6.localdomain6 localhost6
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 10 weeks, 1 day ago 0|i38jzz:
ZooKeeper ZOOKEEPER-2662

Export a metric for txn log sync times

Improvement Resolved Major Fixed Edward Ribeiro Andrew Kyle Purtell Andrew Kyle Purtell 10/Jan/17 21:08   28/Apr/17 12:16 28/Apr/17 11:58   3.5.4, 3.6.0     0 7   In FileTxnLog there is code that records the amount of time required to fsync the txn log in order to warn if that time exceeds a configurable threshold. This information should also be exported as a metric available by JMX so an important aspect of quorum performance can be monitored.

ZooKeeperServerMXBean carries some global latency information for the server process already, seems like a good place to put it if not an entirely new bean for the TxnLog. After ZOOKEEPER-2310 might want to collect the same information for snapshots.
9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
2 years, 46 weeks, 6 days ago 0|i38j3z:
ZooKeeper ZOOKEEPER-2661

It costs about 5055 ms to create Zookeeper object for the first time.

Bug Resolved Major Not A Problem Unassigned Yaohui Wu Yaohui Wu 10/Jan/17 06:20   23/Jan/17 14:46 11/Jan/17 03:58 3.4.6   java client   0 3   See the description below. I create and close ZooKeeper for 10 times. It costs about 5055 ms for the first time.

See attached files for some test code and output.
9223372036854775807 No Perforce job exists for this issue. 2 9223372036854775807
3 years, 8 weeks, 3 days ago 0|i38hon:
ZooKeeper ZOOKEEPER-2660

acceptedEpoch and currentEpoch data inconsistency, ZK process can not start!

Bug Resolved Major Duplicate Unassigned Yongcheng Liu Yongcheng Liu 07/Jan/17 08:14   20/Jan/17 13:03 20/Jan/17 13:03 3.4.6, 3.4.9   quorum   0 3   ZK: 3.4.9 1. currentEpoch is bigger than acceptedEpoch, ZK will throw IOException when start loadDataBase.
2. function bug. In function setAcceptedEpoch and setCurrentEpoch, it is modify memory variable first, then write epoch to file. If write file failed, the memory has been modified.

solution as follow:
for example,

public void setAcceptedEpoch(long e) throws IOException {
acceptedEpoch = e;
writeLongToFile(ACCEPTED_EPOCH_FILENAME, e);
}

need to modify as follow:

public void setAcceptedEpoch(long e) throws IOException {
writeLongToFile(ACCEPTED_EPOCH_FILENAME, e);
acceptedEpoch = e;
}
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 8 weeks, 6 days ago 0|i38eb3:
ZooKeeper ZOOKEEPER-2659

Use log4j2 as a logging framework as log4j 1.X is now deprecated

Wish Resolved Minor Won't Do Pushkar Raste Pushkar Raste Pushkar Raste 06/Jan/17 14:37   30/Jan/19 08:36 17/Mar/17 15:03         1 5 0 600   Zookeeper currently uses {{log4j 1.X}} as the default logging framework. {{log4j 1.X}} is now deprecated http://logging.apache.org/log4j/1.2/

This ticket is to track efforts to move zookeeper to {{log4j2}}
100% 100% 600 0 pull-request-available 9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
2 years, 50 weeks ago 0|i38dcv:
ZooKeeper ZOOKEEPER-2658

Trunk / branch-3.5 build broken.

Task Closed Critical Fixed Michael Han Michael Han Michael Han 03/Jan/17 12:35   17/May/17 23:43 09/Jan/17 17:33 3.5.2, 3.6.0 3.5.3, 3.6.0     0 1   https://builds.apache.org/job/ZooKeeper-trunk-openjdk7/
https://builds.apache.org/job/ZooKeeper_branch35_openjdk7/

The trunk build is broken for over two weeks. It is likely caused by Infrastructure issues.
{noformat}
[ZooKeeper-trunk-openjdk7] $ /home/jenkins/tools/ant/latest/bin/ant -Dtest.output=yes -Dtest.junit.threads=8 -Dtest.junit.output.format=xml -Djavac.target=1.7 clean test-core-java
Error: JAVA_HOME is not defined correctly.
We cannot execute /usr/lib/jvm/java-7-openjdk-amd64//bin/java
Build step 'Invoke Ant' marked build as failure
Recording test results
ERROR: Step ?Publish JUnit test result report? failed: No test report files were found. Configuration error?
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
{noformat}
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 10 weeks, 3 days ago 0|i387lb:
ZooKeeper ZOOKEEPER-2657

Using zookeeper without SASL causes error logging

Improvement Open Major Unresolved Unassigned Aseem Bansal Aseem Bansal 29/Dec/16 05:07   29/Dec/16 05:07   3.4.6       0 2   We are using Kafka which uses zookeeper. But we are not using SASL. So we keep on getting

{noformat}
CRITICAL: Found 32 lines (limit=1/1): (1) 2016-12-16 07:02:14.780 [INFO ] [r] org.apache.zookeeper.ClientCnxn [] - Opening socket connection to server 10.0.1.47/10.0.1.47:2181. Will not attempt to authenticate using SASL (unknown error)
{noformat}

Found http://stackoverflow.com/a/26532778/2235567

Looked and found this based on the above https://svn.apache.org/repos/asf/zookeeper/trunk/src/java/main/org/apache/zookeeper/client/ZooKeeperSaslClient.java

Searched for "Will not attempt to authenticate using SASL" and found the "unknown error".

Can the message be changed so that the word error is not there as it is not really an error?
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 12 weeks ago 0|i383of:
ZooKeeper ZOOKEEPER-2656

Fix ServerConfigTest#testValidArguments test case failures

Test Closed Major Fixed Michael Han Rakesh Radhakrishnan Rakesh Radhakrishnan 22/Dec/16 23:21   31/Mar/17 05:01 06/Jan/17 07:27   3.4.10, 3.5.3, 3.6.0     0 4   This jira to fix ServerConfigTest#testValidArguments test case failure. Reference: https://builds.apache.org/job/ZooKeeper-trunk/3207/testReport/org.apache.zookeeper/ServerConfigTest/testValidArguments/
{code}
Error Message

expected: java.lang.String</data/dir> but was: java.io.File</data/dir>
Stacktrace

junit.framework.AssertionFailedError: expected: java.lang.String</data/dir> but was: java.io.File</data/dir>
at org.apache.zookeeper.ServerConfigTest.testValidArguments(ServerConfigTest.java:48)
{code}
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 10 weeks, 6 days ago 0|i37yvr:
ZooKeeper ZOOKEEPER-2655

Improve NIOServerCnxn#isZKServerRunning to reflect the semantics correctly

Improvement Closed Minor Fixed Rakesh Radhakrishnan Rakesh Radhakrishnan Rakesh Radhakrishnan 22/Dec/16 13:34   17/May/17 23:43 22/Dec/16 17:24   3.5.3, 3.6.0 server   0 4   This jira to improve the semantics of the following internal functions to make it more readable:

# {{NIOServerCnxn#isZKServerRunning()}} => return true if the server is running, false otherwise.
# {{AbstractFourLetterCommand#isZKServerRunning()}} => return true if the server is running, false otherwise.
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 12 weeks, 6 days ago 0|i37ybr:
ZooKeeper ZOOKEEPER-2654

Support Fedora 25: use pkg-config instead of obsolete M4 macros

Bug Patch Available Major Unresolved Olaf Flebbe Olaf Flebbe Olaf Flebbe 22/Dec/16 12:40   22/Dec/16 13:14       build   0 1   While compiling Bigtop on Fedora 25 we found that there is an issue with the autoconf detection of cppunit: See BIGTOP-2642 for error.

Some background regarding the issue can be found here: https://bugzilla.redhat.com/show_bug.cgi?id=1311694

The fedora maintainers encourage use of pkg-config rather crufty *.m4 autoconf magic by only supplying pkg-config files *.pc.

The patch is surprisingly easy but adds the additional requirement for pkg-config which should be available on every well maintained system for ages.

Please see for me proposed patch. Works for me for Fedora 25, Centos 6, MacOSX with HomeBrew.

9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
3 years, 13 weeks ago 0|i37y9j:
ZooKeeper ZOOKEEPER-2653

epoch files do not match snapshots and logs

Bug Open Major Unresolved Unassigned Lasaro Camargos Lasaro Camargos 22/Dec/16 12:03   22/Dec/16 12:03   3.4.9       0 2   Linux 3.10.0-327.el7.x86_64 #1 SMP Thu Nov 19 22:10:57 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux (centos) Hi all.
After shutting zk down and upgrading to centos 7, ZK would not start with exception

Removing file: Dec 19, 2016 10:55:08 PM /hedvig/hpod/log/version-2/log.300ee0308
Removing file: Dec 19, 2016 7:11:23 PM /hedvig/hpod/data/version-2/snapshot.300ee0307
java.lang.RuntimeException: Unable to run quorum server
at org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase(QuorumPeer.java:558)
at org.apache.zookeeper.server.quorum.QuorumPeer.start(QuorumPeer.java:500)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:153)
at com.hedvig.hpod.service.PodnetService$1.run(PodnetService.java:2262)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: The current epoch, 3, is older than the last zxid, 17179871862
at org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase(QuorumPeer.java:539)
... 4 more

All logs are empty, and the following snapshot and commit logs exist

find .
.
./log
./log/version-2
./log/version-2/log.40000010a
./log/version-2/log.300ef712b
./log/version-2/log.300f0659e
./log/version-2/.ignore
./data
./data/version-2
./data/version-2/snapshot.400000109
./data/version-2/currentEpoch
./data/version-2/acceptedEpoch
./data/version-2/snapshot.300ef712a
./data/version-2/snapshot.300f0659d
./data/myid.bak
./data/myid


On other nodes we had the same exception but no commit log deletion.
java.lang.RuntimeException: Unable to run quorum server
at org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase(QuorumPeer.java:558)
at org.apache.zookeeper.server.quorum.QuorumPeer.start(QuorumPeer.java:500)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:153)
at com.hedvig.hpod.service.PodnetService$1.run(PodnetService.java:2262)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: The current epoch, 3, is older than the last zxid, 17179871862
at org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase(QuorumPeer.java:539)

./log
./log/version-2
./log/version-2/log.300f06cfc
./log/version-2/log.300f03890
./log/version-2/.ignore
./data
./data/version-2
./data/version-2/snapshot.300f06cfb
./data/version-2/snapshot.300f06f10
./data/version-2/currentEpoch
./data/version-2/acceptedEpoch
./data/version-2/snapshot.300f0388f
./data/myid.bak
./data/myid


/log
./log/version-2
./log/version-2/log.300f06dbf
./log/version-2/log.300ed96fc
./log/version-2/log.300ef1048
./log/version-2/.ignore
./data
./data/version-2
./data/version-2/snapshot.300f06dbe
./data/version-2/currentEpoch
./data/version-2/acceptedEpoch
./data/version-2/snapshot.300ed96fb
./data/version-2/snapshot.300ef1048
./data/myid.bak
./data/myid

The symptoms look like ZOOKEEPER-1549, but we are running 3.4.9 here.

Any ideas?
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 13 weeks ago 0|i37y7z:
ZooKeeper ZOOKEEPER-2652

Fix HierarchicalQuorumTest.java

Bug Closed Major Fixed Rakesh Radhakrishnan Rakesh Radhakrishnan Rakesh Radhakrishnan 22/Dec/16 01:15   31/Mar/17 05:01 22/Dec/16 01:34 3.4.10 3.4.10     0 4   The commit of ZOOKEEPER-2479 has introduced a compilation error(due to diamond operator usage) in {{branch-3.4}}, which uses {{JDK 1.6}} 9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 12 weeks, 6 days ago 0|i37x4v:
ZooKeeper ZOOKEEPER-2651

Missing src/pom.template in release

Bug Closed Major Fixed Rakesh Radhakrishnan Christopher Tubbs Christopher Tubbs 21/Dec/16 20:47   31/Mar/17 05:01 11/Jan/17 00:28 3.4.9, 3.5.2 3.4.10, 3.5.3, 3.6.0 build   0 6   Trying to build downstream in Fedora, and discovered that the 3.4.9 release tarball is missing the {{src/pom.template}} file. It is present in the {{release-3.4.9}} tag, so I grabbed it from there to patch downstream. 9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 10 weeks, 1 day ago 0|i37wvz:
ZooKeeper ZOOKEEPER-2650

Test Improvement by adding more QuorumPeer Auth related test cases

Test Closed Major Fixed Rakesh Radhakrishnan Rakesh Radhakrishnan Rakesh Radhakrishnan 21/Dec/16 10:11   31/Mar/17 05:01 04/Jan/17 00:30   3.4.10     0 4   This jira to add more test cases to the ZOOKEEPER-1045 feature.
Cases:-
1) Ensemble with auth enabled Observer.
2) Connecting non-auth Observer to auth enabled quorum.
3) Quorum re-election with auth enabled servers.
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 7 weeks, 2 days ago 0|i37vpb:
ZooKeeper ZOOKEEPER-2649

The ZooKeeper do not write in log session ID in which the client has been authenticated.

Improvement Open Trivial Unresolved Alex Zhou Alex Zhou Alex Zhou 19/Dec/16 10:30   05/Feb/20 07:16   3.4.9, 3.5.2 3.7.0, 3.5.8 server   0 2   The ZooKeeper do not write in log session ID in which the client has been authenticated. This occurs for digest and for SASL authentications:

bq. 2016-12-09 15:46:34,808 [myid:] - INFO [SyncThread:0:ZooKeeperServer@673] - Established session 0x158e39a0a960001 with negotiated timeout 30000 for client /0:0:0:0:0:0:0:1:52626
bq. 2016-12-09 15:46:34,838 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:SaslServerCallbackHandler@118] - Successfully authenticated client: authenticationID=bob; authorizationID=bob.
bq. 2016-12-09 15:46:34,848 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:SaslServerCallbackHandler@134] - Setting authorizedID: bob
bq. 2016-12-09 15:46:34,848 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@1024] - adding SASL authorization for authorizationID: bob

bq. 2016-12-13 10:52:54,915 [myid:] - INFO [SyncThread:0:ZooKeeperServer@673] - Established session 0x158f72acaed0001 with negotiated timeout 30000 for client /172.20.97.175:52217
bq. 2016-12-13 10:52:55,070 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:SaslServerCallbackHandler@118] - Successfully authenticated client: authenticationID=ufm@BILLAB.RU; authorizationID=ufm@BILLAB.RU.
bq. 2016-12-13 10:52:55,075 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:SaslServerCallbackHandler@134] - Setting authorizedID: ufm@BILLAB.RU
bq. 2016-12-13 10:52:55,075 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@1024] - adding SASL authorization for authorizationID: ufm@BILLAB.RU

bq. 2016-12-19 17:43:01,395 [myid:] - INFO [SyncThread:0:ZooKeeperServer@673] - Established session 0x158fd72521f0000 with negotiated timeout 30000 for client /172.20.97.175:57633
bq. 2016-12-19 17:45:53,497 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@952] - got auth packet /172.20.97.175:57633
bq. 2016-12-19 17:45:53,508 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@986] - auth success /172.20.97.175:57633

So, it is difficult to determine which client made changes.
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 13 weeks, 2 days ago 0|i37ry7:
ZooKeeper ZOOKEEPER-2648

Container node never gets deleted if it never had children

Bug Resolved Major Not A Bug Unassigned Hadriel Kaplan Hadriel Kaplan 18/Dec/16 13:05   01/Aug/17 14:35 23/Jan/17 17:11 3.5.0   server   0 4   If a client creates a Container node, but does not also create a child within that Container, the Container will never be deleted. This may seem like a bug in the client for not subsequently creating a child, but we can't assume the client remains connected, or that the client didn't just change its mind (due to some recipe being canceled, for example).

The bug is in ContainerManager.getCandidates(), which only considers a node a candidate if its Cversion > 0. The comments indicate this was done intentionally, to avoid a race condition whereby the Container was created right before a cleaning period, and would get cleaned up before the child could be created - so to avoid that the check is performed to verify the Cversion > 0.

Instead, I propose that if the Cversion is 0 but the Ctime is more than a checkIntervalMs old, then it be deleted. In other words, if the Container node has been around for a whole cleaning round already and no child has been created since, then go ahead and clean it up.

I can provide a patch if others agree with such a change.



9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 8 weeks, 3 days ago 0|i37qvz:
ZooKeeper ZOOKEEPER-2647

Fix TestReconfigServer.cc

Bug Closed Blocker Fixed Flavio Paiva Junqueira Flavio Paiva Junqueira Flavio Paiva Junqueira 16/Dec/16 06:33   17/May/17 23:43 05/Jan/17 12:57   3.5.3, 3.6.0     0 5   The commit of ZOOKEEPER-761 has introduced a compilation error in one of the test cases. It is a pretty straightforward fix. 9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 11 weeks ago 0|i37osn:
ZooKeeper ZOOKEEPER-2646

Java target in branch 3.4 doesn't match documentation

Bug Closed Major Fixed Flavio Paiva Junqueira Flavio Paiva Junqueira Flavio Paiva Junqueira 15/Dec/16 17:22   31/Mar/17 05:01 21/Dec/16 23:48 3.4.9 3.4.10     0 5   Need to update build.xml 1.5->1.6. 9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 7 weeks ago 0|i37nz3:
ZooKeeper ZOOKEEPER-2645

If embedded QuorumPeerMain is started with Properties no backupOldConfig should be done

Bug Open Major Unresolved Unassigned Tom Pijl Tom Pijl 15/Dec/16 02:36   15/Dec/16 03:04   3.5.2   server   2 4   When starting an embedded _QuorumPeerMain_ by executing the _runFromConfig()_ and providing _QuorumPeerConfig_ properties:
{code}standaloneEnabled=false
initLimit=5
syncLimit=2
clientPort=4101
server.1=nlbantpijl01.infor.com:2101:3101:participant;4101
dataDir=/Storage/zookeeper/server001{code}
an NullPointerException is thrown in the _QuorumPeerConfig_ class in the method _backupOldConfig()_ because the property configFileStr is null.

A check must be made at the start of the method _backupOldConfig()_ if the property _configFileStr_ is null. If so just exit the method. In the embedded mode there is no config file, so no need to create a backup.
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 14 weeks ago 0|i37mnz:
ZooKeeper ZOOKEEPER-2644

contrib/rest does not include rest.sh when packaged

Improvement Open Minor Unresolved Unassigned Minoru Osuka Minoru Osuka 12/Dec/16 09:21   30/Jan/19 08:16   3.4.9, 3.5.2   contrib   0 3 0 600   contrib/rest does not include rest.sh when packaged. I propose to add rest.sh into tar.gz that it make ZooKeeper REST easier to use. 100% 100% 600 0 pull-request-available 9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
3 years, 13 weeks, 5 days ago 0|i37hen:
ZooKeeper ZOOKEEPER-2643

Configurable SSLContext

Wish Patch Available Major Unresolved Unassigned George Goddard George Goddard 09/Dec/16 13:47   09/Dec/16 13:59           0 1   Being able to configure the SSLContext in X509Util.java, ZKConfig.java and NettyServerCnxnFactory.java would add flexibility to use cipher suites other than TLSv1. 9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
3 years, 14 weeks, 6 days ago 0|i37eyn:
ZooKeeper ZOOKEEPER-2642

ZooKeeper reconfig API backward compatibility fix

Bug Closed Blocker Fixed Jordan Zimmerman Jordan Zimmerman Jordan Zimmerman 07/Dec/16 06:23   30/Jan/19 11:16 11/Feb/17 18:34 3.5.2 3.5.3, 3.6.0 c client, java client   0 7 0 1800   ZOOKEEPER-2014 moved the reconfig() methods into a new class, ZooKeeperAdmin. It appears this was done to document that these are methods have access restrictions. However, this change breaks Apache Curator (and possibly other clients). Curator APIs will have to be changed and/or special methods need to be added. A breaking change of this kind should only be done when the benefit is overwhelming. In this case, the same information can be conveyed with documentation and possibly a deprecation notice.

Revert the creation of the ZooKeeperAdmin class and move the reconfig() methods back to the ZooKeeper class with additional documentation.
100% 100% 1800 0 pull-request-available 9223372036854775807 No Perforce job exists for this issue. 7 9223372036854775807
2 years, 48 weeks, 3 days ago 0|i37afb:
ZooKeeper ZOOKEEPER-2641

AvgRequestLatency metric improves to be more accurate

Improvement Resolved Minor Fixed maoling Allen Chan Allen Chan 04/Dec/16 23:57   20/Jan/19 06:55 07/Jan/19 08:39 3.4.7, 3.4.9 3.6.0     1 5 0 15600   I posted a thread on the mailing list about finding AvgRequestLatency metric to be 0 all the time. I believe this is a valuable metric because it is useful to baseline performance of ZK and know when something is going wrong.

Another user (Arshad Mohammad) wrote up these notes. I am not a developer so i do not have ability to patch this. Filing this so hopefully someone with developer abilities can add this improvement.

"I find two reason whys AvgRequestLatency is almost all the time 0
1) Ping requests are counted the most:
AvgRequestLatency is calculated as
AvgRequestLatency=totalLatency/count

Ping requests come very often and complete very fast, these request add
nothing to totalLatency but add one to count.

2) Wrong data type is chosen to store AvgRequestLatency:
AvgRequestLatency is calculated and store as the long value instead of
double vlaue.
In my opinion ZooKeeper code should be modified to improve this metrics

i) Ping request should be ignored while recording the statistics or at
least should be configurable whether to ignore or not. If ping request is
not counted even other metrics will be more meaningful.

ii) AvgRequestLatency should be of double type"
100% 100% 15600 0 pull-request-available 9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
1 year, 10 weeks, 3 days ago 0|i375bb:
ZooKeeper ZOOKEEPER-2640

fix test coverage for single threaded C-API

Test Open Major Unresolved Unassigned Benjamin Reed Benjamin Reed 04/Dec/16 19:25   04/Dec/16 19:28       c client, tests   0 1   the tests for the C-API are mostly for the multithreaded API. we need to get better coverage for the single threaded API. 9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 15 weeks, 4 days ago 0|i3750v:
ZooKeeper ZOOKEEPER-2639

Port Quorum Peer mutual authentication SASL feature to branch-3.5 and trunk

Task Open Critical Unresolved Rakesh Radhakrishnan Rakesh Radhakrishnan Rakesh Radhakrishnan 30/Nov/16 01:04   14/Dec/19 06:08     3.7.0 quorum, security   1 6   ZOOKEEPER-2792, ZOOKEEPER-2793, ZOOKEEPER-2794, ZOOKEEPER-2850, ZOOKEEPER-2851, ZOOKEEPER-2935 ZooKeeper server-server mutual authentication is implemented in {{branch-3.4}} using ZOOKEEPER-1045 jira. The feature code is not directly portable to other branches due to code difference. This jira can be used to "forward" port the code changes to {{branch-3.5}} and {{trunk}}. 9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
1 year, 17 weeks ago 0|i36y1j:
ZooKeeper ZOOKEEPER-2638

ZooKeeper should log which serverCnxnFactory is used during startup

Improvement Resolved Major Fixed Abraham Fine Abraham Fine Abraham Fine 29/Nov/16 18:34   18/Apr/17 21:12 18/Apr/17 19:19 3.5.2 3.5.4, 3.6.0     0 5   9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
2 years, 48 weeks, 1 day ago 0|i36xlj:
ZooKeeper ZOOKEEPER-2637

ZOOKEEPER-2135 NettyNettySuiteHammerTest and NettyNettySuiteTest failures

Sub-task Open Major Unresolved Vineet Ghatge Amita Chaudhary Amita Chaudhary 28/Nov/16 08:26   19/Mar/17 13:14   3.6.0   tests   0 4   rhel ppc64le I am getting test failures related to Netty:
[junit] Running org.apache.zookeeper.test.NettyNettySuiteHammerTest
[junit] Running org.apache.zookeeper.test.NettyNettySuiteHammerTest
[junit] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0 sec
[junit] Test org.apache.zookeeper.test.NettyNettySuiteHammerTest FAILED (crashed)
[junit] Running org.apache.zookeeper.test.NettyNettySuiteTest
[junit] Tests run: 101, Failures: 0, Errors: 26, Skipped: 0, Time elapsed: 247.238 sec
[junit] Test org.apache.zookeeper.test.NettyNettySuiteTest FAILED
[junit] Running org.apache.zookeeper.test.NioNettySuiteHammerTest
[junit] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 75.519 sec
on machine rhel, ppc64le. for master branch.
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 4 days ago 0|i36uef:
ZooKeeper ZOOKEEPER-2636

Fix C build break.

Bug Closed Blocker Fixed Michael Han Michael Han Michael Han 24/Nov/16 16:28   17/May/17 23:44 25/Nov/16 01:10 3.5.3 3.5.3, 3.6.0 jute   0 5   C client build is broken after ZOOKEEPER-2628 is merged in. After a little debug, I find out that the build is broken because the zookeeper.jute.h and zookeeper.jute.c are not completely generated.

* The culprit is the code change introduced in ZOOKEEPER-2628, where we wraps {code}JRecord.genCCode{code} with a try / catch / finally block and the file writers were prematurely closed in finally block which prevents remaining of the zookeeper.jute.h/c file being generated.

* The fix to {code}JRecord.genCCode{code} in ZOOKEEPER-2628 was made because a find bug warning was directly associated with the code. Due to the subtlety of the file writer ownership, we did not capture the issue during code review.

* The build break was not captured in pre-commit builds as well ([an example|https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/72//console]), where we get all tests passed including C client tests. I suspect we might have another bug with cached generated files that should be regenerated but we don't - need more investigation on this one.

* The fix is simple by revert the change to this specific method. Findbug does not complain anymore because the previous warning that appertain to this code block was fixed at the call site of {code}JRecord.genCCode{code}. So by reverting the change we still have zero find bug warnings.
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 16 weeks, 6 days ago 0|i36rhr:
ZooKeeper ZOOKEEPER-2635

Regenerate documentation

Bug Closed Blocker Fixed Michael Han Flavio Paiva Junqueira Flavio Paiva Junqueira 24/Nov/16 11:38   17/May/17 23:44 21/Mar/17 15:10   3.5.3, 3.6.0 documentation   0 5   Some recent commits did not regenerate the documentation even though they had documentation changes, we need to do it before releasing. 9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 2 days ago 0|i36rbz:
ZooKeeper ZOOKEEPER-2634

null data in zknode data

Bug Open Major Unresolved Unassigned nayeem nayeem 20/Nov/16 08:01   26/Aug/18 23:09   3.4.5   java client   0 3   linux zookeeper 3.4.5 We can create zk node with null data as given bellow.
ZkConnect connector = new ZkConnect();
ZooKeeper zk = connector.connect("host:port");
String newNode = "/nayeemDate3";
String strdata = String.valueOf('\u0000');
connector.createNode(newNode, strdata.getBytes());

When we get the data for the zknode
2016-11-17 23:55:48,926 [myid:] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:5181:NIOServerCnxn@349] - caught end of stream exception
EndOfStreamException: Unable to read additional data from client sessionid 0x1585061acbd0613, likely client has closed socket
at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
at java.lang.Thread.run(Thread.java:745)
2016-11-17 23:55:48,926 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:5181:NIOServerCnxn@1001] - Closed socket connection for client /10.10.72.93:48005 which had sessionid 0x1585061acbd0613

To resolve the issue workaround is to delete the zknode,

is it the right behaviour or is this a bug.
data from zkcli
[zk: 10.10.72.93:5181(CONNECTED) 1] ls /nayeemDate3
[]
[zk: 10.10.72.93:5181(CONNECTED) 2] get /nayeemDate3
null
cZxid = 0xdc47
ctime = Fri Nov 18 13:29:43 IST 2016
mZxid = 0xdc47
mtime = Fri Nov 18 13:29:43 IST 2016
pZxid = 0xdc47
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 0
numChildren = 0
[zk: 10.10.72.93:5181(CONNECTED) 3] stat /nayeemDate3
cZxid = 0xdc47
ctime = Fri Nov 18 13:29:43 IST 2016
mZxid = 0xdc47
mtime = Fri Nov 18 13:29:43 IST 2016
pZxid = 0xdc47
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 0
numChildren = 0
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
Important
1 year, 29 weeks, 3 days ago 0|i36jo7:
ZooKeeper ZOOKEEPER-2633

Build failure in contrib/zkfuse with gcc 6.x

Bug Closed Minor Fixed Raghavendra Prabhu Raghavendra Prabhu Raghavendra Prabhu 17/Nov/16 06:11   31/Mar/17 05:01 22/Jan/17 19:43   3.4.10, 3.5.3, 3.6.0 contrib-zkfuse   0 6   gcc --version
gcc (GCC) 6.2.1 20160830
Copyright (C) 2016 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.


g++ --version
g++ (GCC) 6.2.1 20160830
Copyright (C) 2016 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

CFLAGS, CXXFLAGS, and LDFLAGS are unset, hence default.

uname -a
Linux lative 4.8.8-1-ARCH #1 SMP PREEMPT Tue Nov 15 08:25:24 CET 2016 x86_64 GNU/Linux
The build in contrib/zkfuse fails with

{noformat}

make
(CDPATH="${ZSH_VERSION+.}:" && cd . && /bin/sh /home/raghu/zookeeper/src/contrib/zkfuse/missing autoheader)
rm -f stamp-h1
touch config.h.in
cd . && /bin/sh ./config.status config.h
config.status: creating config.h
config.status: config.h is unchanged
make all-recursive
make[1]: Entering directory '/home/raghu/zookeeper/src/contrib/zkfuse'
Making all in src
make[2]: Entering directory '/home/raghu/zookeeper/src/contrib/zkfuse/src'
g++ -DHAVE_CONFIG_H -I. -I.. -I/home/raghu/zookeeper/src/contrib/zkfuse/../../c/include -I/home/raghu/zookeeper/src/contrib/zkfuse/../../c/generated -I../include -I/usr/include -D_FILE_OFFSET_BITS=64 -D_REENTRANT -march=x86-64 -mtune=generic -O2 -pipe -fstack-protector-strong -MT zkfuse.o -MD -MP -MF .deps/zkfuse.Tpo -c -o zkfuse.o zkfuse.cc
g++ -DHAVE_CONFIG_H -I. -I.. -I/home/raghu/zookeeper/src/contrib/zkfuse/../../c/include -I/home/raghu/zookeeper/src/contrib/zkfuse/../../c/generated -I../include -I/usr/include -D_FILE_OFFSET_BITS=64 -D_REENTRANT -march=x86-64 -mtune=generic -O2 -pipe -fstack-protector-strong -MT zkadapter.o -MD -MP -MF .deps/zkadapter.Tpo -c -o zkadapter.o zkadapter.cc
In file included from zkadapter.h:34:0,
from zkadapter.cc:24:
event.h:216:9: error: reference to ‘shared_ptr’ is ambiguous
shared_ptr<AbstractEventWrapper> m_eventWrapper;
^~~~~~~~~~
In file included from /usr/include/boost/throw_exception.hpp:42:0,
from /usr/include/boost/smart_ptr/shared_ptr.hpp:27,
from /usr/include/boost/shared_ptr.hpp:17,
from event.h:30,
from zkadapter.h:34,
from zkadapter.cc:24:
/usr/include/boost/exception/exception.hpp:148:11: note: candidates are: template<class T> class boost::shared_ptr
class shared_ptr;
^~~~~~~~~~
In file included from /usr/include/c++/6.2.1/bits/shared_ptr.h:52:0,
from /usr/include/c++/6.2.1/memory:82,
from /usr/include/boost/config/no_tr1/memory.hpp:21,
from /usr/include/boost/smart_ptr/shared_ptr.hpp:23,
from /usr/include/boost/shared_ptr.hpp:17,
from event.h:30,
from zkadapter.h:34,
from zkadapter.cc:24:
/usr/include/c++/6.2.1/bits/shared_ptr_base.h:343:11: note: template<class _Tp> class std::shared_ptr
class shared_ptr;
^~~~~~~~~~
In file included from zkadapter.h:34:0,
from zkadapter.cc:24:
event.h: In constructor ‘zkfuse::GenericEvent::GenericEvent(int, zkfuse::AbstractEventWrapper*)’:
event.h:189:27: error: class ‘zkfuse::GenericEvent’ does not have any field named ‘m_eventWrapper’
m_type(type), m_eventWrapper(eventWrapper) {
^~~~~~~~~~~~~~
event.h: In member function ‘void* zkfuse::GenericEvent::getEvent() const’:
event.h:204:41: error: ‘m_eventWrapper’ was not declared in this scope
void *getEvent() const { return m_eventWrapper->getWrapee(); }
^~~~~~~~~~~~~~
In file included from zkadapter.h:34:0,
from zkfuse.cc:54:
event.h:216:9: error: reference to ‘shared_ptr’ is ambiguous
shared_ptr<AbstractEventWrapper> m_eventWrapper;
^~~~~~~~~~
In file included from /usr/include/boost/throw_exception.hpp:42:0,
from /usr/include/boost/smart_ptr/detail/shared_count.hpp:27,
from /usr/include/boost/smart_ptr/weak_ptr.hpp:17,
from /usr/include/boost/weak_ptr.hpp:16,
from zkfuse.cc:50:
/usr/include/boost/exception/exception.hpp:148:11: note: candidates are: template<class T> class boost::shared_ptr
class shared_ptr;
^~~~~~~~~~
In file included from /usr/include/c++/6.2.1/bits/shared_ptr.h:52:0,
from /usr/include/c++/6.2.1/memory:82,
from /usr/include/boost/smart_ptr/weak_ptr.hpp:16,
from /usr/include/boost/weak_ptr.hpp:16,
from zkfuse.cc:50:
/usr/include/c++/6.2.1/bits/shared_ptr_base.h:343:11: note: template<class _Tp> class std::shared_ptr
class shared_ptr;
^~~~~~~~~~
In file included from zkadapter.h:34:0,
from zkfuse.cc:54:
event.h: In constructor ‘zkfuse::GenericEvent::GenericEvent(int, zkfuse::AbstractEventWrapper*)’:
event.h:189:27: error: class ‘zkfuse::GenericEvent’ does not have any field named ‘m_eventWrapper’
m_type(type), m_eventWrapper(eventWrapper) {
^~~~~~~~~~~~~~
event.h: In member function ‘void* zkfuse::GenericEvent::getEvent() const’:
event.h:204:41: error: ‘m_eventWrapper’ was not declared in this scope
void *getEvent() const { return m_eventWrapper->getWrapee(); }
^~~~~~~~~~~~~~
zkadapter.cc: In member function ‘bool zk::ZooKeeperAdapter::deleteNode(const string&, bool, int)’:
zkadapter.cc:676:52: error: no matching function for call to ‘zk::ZooKeeperAdapter::getNodeChildren(std::vector<std::__cxx11::basic_string<char> >&, const string&, bool)’
getNodeChildren( nodeList, path, false );
^
In file included from zkadapter.cc:24:0:
zkadapter.h:440:14: note: candidate: void zk::ZooKeeperAdapter::getNodeChildren(std::vector<std::__cxx11::basic_string<char> >&, const string&, zk::ZKEventListener*, void*)
void getNodeChildren(vector<string> &children,
^~~~~~~~~~~~~~~
zkadapter.h:440:14: note: no known conversion for argument 3 from ‘bool’ to ‘zk::ZKEventListener* {aka zkfuse::EventListener<zk::ZKWatcherEvent>*}’
make[2]: *** [Makefile:310: zkadapter.o] Error 1
make[2]: *** Waiting for unfinished jobs....
make[2]: *** [Makefile:310: zkfuse.o] Error 1
make[2]: Leaving directory '/home/raghu/zookeeper/src/contrib/zkfuse/src'
make[1]: *** [Makefile:352: all-recursive] Error 1
make[1]: Leaving directory '/home/raghu/zookeeper/src/contrib/zkfuse'
make: *** [Makefile:293: all] Error 2


=================================================================================================================


make
make all-recursive
make[1]: Entering directory '/home/raghu/zookeeper/src/contrib/zkfuse'
Making all in src
make[2]: Entering directory '/home/raghu/zookeeper/src/contrib/zkfuse/src'
g++ -DHAVE_CONFIG_H -I. -I.. -I/home/raghu/zookeeper/src/contrib/zkfuse/../../c/include -I/home/raghu/zookeeper/src/contrib/zkfuse/../../c/generated -I../include -I/usr/include -D_FILE_OFFSET_BITS=64 -D_REENTRANT -march=x86-64 -mtune=generic -O2 -pipe -fstack-protector-strong -MT zkfuse.o -MD -MP -MF .deps/zkfuse.Tpo -c -o zkfuse.o zkfuse.cc
g++ -DHAVE_CONFIG_H -I. -I.. -I/home/raghu/zookeeper/src/contrib/zkfuse/../../c/include -I/home/raghu/zookeeper/src/contrib/zkfuse/../../c/generated -I../include -I/usr/include -D_FILE_OFFSET_BITS=64 -D_REENTRANT -march=x86-64 -mtune=generic -O2 -pipe -fstack-protector-strong -MT zkadapter.o -MD -MP -MF .deps/zkadapter.Tpo -c -o zkadapter.o zkadapter.cc
zkadapter.cc: In member function ‘bool zk::ZooKeeperAdapter::deleteNode(const string&, bool, int)’:
zkadapter.cc:676:52: error: no matching function for call to ‘zk::ZooKeeperAdapter::getNodeChildren(std::vector<std::__cxx11::basic_string<char> >&, const string&, bool)’
getNodeChildren( nodeList, path, false );
^
In file included from zkadapter.cc:24:0:
zkadapter.h:440:14: note: candidate: void zk::ZooKeeperAdapter::getNodeChildren(std::vector<std::__cxx11::basic_string<char> >&, const string&, zk::ZKEventListener*, void*)
void getNodeChildren(vector<string> &children,
^~~~~~~~~~~~~~~
zkadapter.h:440:14: note: no known conversion for argument 3 from ‘bool’ to ‘zk::ZKEventListener* {aka zkfuse::EventListener<zk::ZKWatcherEvent>*}’
make[2]: *** [Makefile:310: zkadapter.o] Error 1
make[2]: *** Waiting for unfinished jobs....
mv -f .deps/zkfuse.Tpo .deps/zkfuse.Po
make[2]: Leaving directory '/home/raghu/zookeeper/src/contrib/zkfuse/src'
make[1]: *** [Makefile:352: all-recursive] Error 1
make[1]: Leaving directory '/home/raghu/zookeeper/src/contrib/zkfuse'
make: *** [Makefile:293: all] Error 2

{noformat}

in two different places.

Fixed here: https://github.com/ronin13/zookeeper/commit/726a8eda08e4022fcbcb0581ec2650e07e39910b
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 8 weeks, 1 day ago 0|i36g6n:
ZooKeeper ZOOKEEPER-2632

Add option to inform JIRA_PASSWORD at CLI prompt

Improvement Resolved Trivial Fixed Edward Ribeiro Edward Ribeiro Edward Ribeiro 14/Nov/16 16:28   24/Nov/16 11:14 18/Nov/16 17:07   3.6.0     0 3   Adds the option to prompt for the JIRA password if JIRA_USERNAME is set, but JIRA_PASSWORD is not. Also, asks if the user wants to continue the merge process if the python jira lib is not installed. 9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 17 weeks ago
Reviewed
0|i36b67:
ZooKeeper ZOOKEEPER-2631

Make issue extraction in the git pull request script more robust

Improvement Resolved Major Fixed Flavio Paiva Junqueira Flavio Paiva Junqueira Flavio Paiva Junqueira 11/Nov/16 09:11   30/Jan/19 08:08 12/Nov/16 22:55 3.6.0   build   0 4 0 600   The QA build is failing for some pull requests because the issue title isn't following the expected format. The issue extraction right now is a bit fragile, so this is to fix the issue. 100% 100% 600 0 pull-request-available 9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 18 weeks, 4 days ago 0|i3682f:
ZooKeeper ZOOKEEPER-2630

Use interface type instead of implementation type when appropriate.

Improvement Resolved Trivial Fixed Tamas Penzes Michael Han Michael Han 10/Nov/16 12:19   31/Jan/19 17:13 11/Sep/17 12:55   3.6.0     0 5 0 2400   There are a couple of places in code base where we declare a field / variable as implementation type (i.e. HashMap, HashSet) instead of interface type (i.e. Map, Set), while in other places we do the opposite by declaring as interface type. A quick check indicates that most if not all of these places could be updated so we have a consistent style over the code base (prefer using interface type), which is also a good coding style to stick per best practice.

See more info on https://github.com/apache/zookeeper/pull/102
100% 100% 2400 0 newbie, pull-request-available, refactoring 9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
2 years, 22 weeks, 6 days ago 0|i366of:
ZooKeeper ZOOKEEPER-2629

Clean up git pull request QA script

Improvement Open Minor Unresolved Flavio Paiva Junqueira Flavio Paiva Junqueira Flavio Paiva Junqueira 06/Nov/16 14:16   06/Nov/16 14:17           0 1   We have introduced a script for QA of pull requests on github in ZOOKEEPER-2624. There is some cleanup left to do on the script, e.g., indentation, consistency of brackets, etc. This jira to do this clean-up. 9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 19 weeks, 4 days ago 0|i35zbb:
ZooKeeper ZOOKEEPER-2628

Investigate and fix findbug warnings

Bug Closed Major Fixed Michael Han Michael Han Michael Han 04/Nov/16 20:49   17/May/17 23:44 24/Nov/16 11:21 3.5.2 3.5.3, 3.6.0     0 6   Findbug tool used by Jenkins bot is upgraded to 3.0.1 from 2.0.3 according to Infra team, and this leads to 20 new warnings produced by findbug. The warning reports can be found on [pre commit builds|https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/] with build number >= 3513. These warnings need to be triaged and fixed if they are legitimate.
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 17 weeks ago 0|i35x0n:
ZooKeeper ZOOKEEPER-2627

Remove ZRWSERVERFOUND from C client and replace handle_error with something more semantically explicit for r/w server reconnect.

Bug Closed Major Fixed Michael Han Michael Han Michael Han 03/Nov/16 16:52   17/May/17 23:44 03/Dec/16 12:17 3.5.2 3.5.3 c client   0 5   While working on ZOOKEEPER-2014, I noticed a discrepancy between Java and C client regarding the error codes definition. There is a {noformat}ZRWSERVERFOUND = -122{noformat} definition in C client which is not present in Java client's KeeperException.Code definitions.

This discrepancy was introduced by ZOOKEEPER-827, where the C client logic was simulating the Java client's logic when doing a read/write server search while client is in read only mode. Once client finds a valid read/write server, client will try to disconnect and reconnect with this read/write server, as we always prefer r/w server in ro mode. The way Java client is doing this disconnect/reconnect process is by throwing a RWServerFoundException (instead of a KeeperException) to set the client in disconnected state, then wait for client reconnect with r/w server address set before throwing the exception. C client did similar but instead of having an explicitly disconnect / clean up routine, the client was relying on handle_error to do the job where ZRWSERVERFOUND was introduced.

I propose we remove ZRWSERVERFOUND error code from C client and use an explicit routine instead of handle_error when we do r/w server search in C client for two reasons:
* ZRWSERVERFOUND is not something ZK client users would need to know. It's a pure implementation detail that's used to alter the connection state of the client, and ZK client users have no desire nor need to handle such errors, as R/W server scanning and connect is handled transparently by ZK client library.
* To maintain consistency between Java and C client regarding error codes definition. Without removing this from C client, we would need to replace RWServerFoundException in Java client with a new KeeperException, and again with the reason mentioned above, we don't need a KeeperException for this because such implementation detail does not have to be exposed to end users (unless, we provided alternative for users to opt-out automate R/W server switching when in read only mode which we don't.).
9223372036854775807 No Perforce job exists for this issue. 2 9223372036854775807
3 years, 15 weeks, 5 days ago 0|i35tv3:
ZooKeeper ZOOKEEPER-2626

log4j.properties don't get respected

Bug Open Major Unresolved Unassigned Arne Bachmann Arne Bachmann 02/Nov/16 09:04   26/Jan/18 11:51   3.5.2   scripts   0 4   Linux vagrant-ubuntu-trusty-32 3.13.0-100-generic #147-Ubuntu SMP Tue Oct 18 16:49:53 UTC 2016 i686 i686 i686 GNU/Linux I put the log4j.properties into the conf folder, plus a symlink to the base zookeeper folder, as described in the documentation.
Neither of them seem to be picked up, as my rolling logger is not recognized (no logs created), and also bin/zkServer.sh print-cmd shows wrong logger configuration. Is that a problem of the start script or did I put the properties file into the wrong place?

Note however, that also my additional java command-line options (from JAVA_TOOL_OPTIONS) don't get picked up by the start script, as can be seen by ps aux | grep java (e.g. -Xmx1000m instead of -Xmx500 as I defined it).
The script's refer to a lot of environment variables that aren't explained in the documentation and nowhere defined; I can't get it to run.
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
2 years, 7 weeks, 6 days ago 0|i35q6v:
ZooKeeper ZOOKEEPER-2625

zkServer.sh creates PID file in the folder data?/ instead of data/

Bug Open Minor Unresolved Unassigned Arne Bachmann Arne Bachmann 02/Nov/16 08:59   19/Feb/19 16:58   3.5.2   scripts   0 4 0 5400   Linux vagrant-ubuntu-trusty-32 3.13.0-100-generic #147-Ubuntu SMP Tue Oct 18 16:49:53 UTC 2016 i686 i686 i686 GNU/Linux I provision a vagrant vm that installs zookeeper into /home/vagrant/zk and adjusts all owner and read/write rights.
With the vagrant user, I start zookeeper as bin/zkServer.sh start /vagrant/data/zoo.cfg
However, the folder data? (or data^M) gets created with the PID inside, instead of putting it into the data folder, which contains the version-2 folder.
Since I'm using the official start scripts, I'm at a loss.
Also, the data? folder comes with root:root ownership, which is strange, as zKServer.sh is executed from the vagrant user.
100% 100% 5400 0 build, pull-request-available 9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 18 weeks, 5 days ago 0|i35q67:
ZooKeeper ZOOKEEPER-2624

Add test script for pull requests

Improvement Resolved Major Fixed Flavio Paiva Junqueira Flavio Paiva Junqueira Flavio Paiva Junqueira 31/Oct/16 08:12   30/Jan/19 08:07 06/Nov/16 15:44     scripts   0 4 0 600   We need a script similar to {{test-patch.sh}} to handle QA builds for pull requests. 100% 100% 600 0 pull-request-available 9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 19 weeks, 4 days ago 0|i35mav:
ZooKeeper ZOOKEEPER-2623

CheckVersion outside of Multi causes NullPointerException

Bug Open Minor Unresolved Unassigned Diego Ongaro Diego Ongaro 27/Oct/16 22:29   03/Nov/16 09:54           0 4   I wasn't sure if check version (opcode 13) was permitted outside of a multi op, so I tried it. My server crashed with a NullPointerException and became unusable until restarted. I guess it's not allowed, but perhaps the server should handle this more gracefully?

Here are the server logs:
{noformat}
Accepted socket connection from /0:0:0:0:0:0:0:1:51737
Session establishment request from client /0:0:0:0:0:0:0:1:51737 client's lastZxid is 0x0
Connection request from old client /0:0:0:0:0:0:0:1:51737; will be dropped if server is in r-o mode
Client attempting to establish new session at /0:0:0:0:0:0:0:1:51737
:Fsessionid:0x10025651faa0000 type:createSession cxid:0x0 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a
Processing request:: sessionid:0x10025651faa0000 type:createSession cxid:0x0 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a
Got zxid 0x60000065e expected 0x1
Creating new log file: log.60000065e
Committing request:: sessionid:0x10025651faa0000 type:createSession cxid:0x0 zxid:0x60000065e txntype:-10 reqpath:n/a
Processing request:: sessionid:0x10025651faa0000 type:createSession cxid:0x0 zxid:0x60000065e txntype:-10 reqpath:n/a
:Esessionid:0x10025651faa0000 type:createSession cxid:0x0 zxid:0x60000065e txntype:-10 reqpath:n/a
sessionid:0x10025651faa0000 type:createSession cxid:0x0 zxid:0x60000065e txntype:-10 reqpath:n/a
Add a buffer to outgoingBuffers, sk sun.nio.ch.SelectionKeyImpl@28e9f397 is valid: true
Established session 0x10025651faa0000 with negotiated timeout 20000 for client /0:0:0:0:0:0:0:1:51737
:Fsessionid:0x10025651faa0000 type:check cxid:0x1 zxid:0xfffffffffffffffe txntype:unknown reqpath:/
Processing request:: sessionid:0x10025651faa0000 type:check cxid:0x1 zxid:0xfffffffffffffffe txntype:unknown reqpath:/
Processing request:: sessionid:0x10025651faa0000 type:check cxid:0x1 zxid:0xfffffffffffffffe txntype:unknown reqpath:/
Exception causing close of session 0x10025651faa0000: Connection reset by peer
:Esessionid:0x10025651faa0000 type:check cxid:0x1 zxid:0xfffffffffffffffe txntype:unknown reqpath:/
IOException stack trace
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:197)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:320)
at org.apache.zookeeper.server.NIOServerCnxnFactory$IOWorkRequest.doWork(NIOServerCnxnFactory.java:530)
at org.apache.zookeeper.server.WorkerService$ScheduledWorkRequest.run(WorkerService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Unexpected exception
java.lang.NullPointerException
at org.apache.zookeeper.server.ZKDatabase.addCommittedProposal(ZKDatabase.java:252)
at org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:127)
at org.apache.zookeeper.server.quorum.CommitProcessor$CommitWorkRequest.doWork(CommitProcessor.java:362)
at org.apache.zookeeper.server.WorkerService$ScheduledWorkRequest.run(WorkerService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Committing request:: sessionid:0x10025651faa0000 type:error cxid:0x1 zxid:0x60000065f txntype:-1 reqpath:n/a
Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id1,name1=replica.1,name2=Follower,name3=Connections,name4="0:0:0:0:0:0:0:1",name5=0x10025651faa0000]
Exception thrown by downstream processor, unable to continue.
CommitProcessor exited loop!
Closed socket connection for client /0:0:0:0:0:0:0:1:51737 which had sessionid 0x10025651faa0000
{noformat}

And here's a one-liner to repro, which does a ConnectRequest followed by a {{CheckVersion(path="/", version=89235}}}:
{noformat}
echo AAAALAAAAAAAAAAAAAAAAAAAJxAAAAAAAAAAAAAAABAAAAAAAAAAAAAAAAAAAAAAAAAAEQAAAAEAAAANAAAAAS8AAVyT | base64 --decode | nc localhost 2181 >/dev/null
{noformat}

This is against master as of a couple of weeks ago (f78061a). I haven't checked to see which versions are affected.
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 20 weeks ago 0|i35itj:
ZooKeeper ZOOKEEPER-2622

ZooTrace.logQuorumPacket does nothing

Bug Closed Trivial Fixed Flavio Paiva Junqueira Flavio Paiva Junqueira Flavio Paiva Junqueira 26/Oct/16 10:17   23/Jan/20 13:16 28/Jan/17 00:52   3.4.10, 3.5.3, 3.6.0     0 7   The method simply returns and there is some code commented out:

{code}
// if (isTraceEnabled(log, mask)) {
// logTraceMessage(LOG, mask, direction + " "
// + FollowerHandler.packetToString(qp));
// }
{code}

There are calls to this trace method, so I think we should fix it.
newbie 9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 7 weeks, 5 days ago 0|i35f6v:
ZooKeeper ZOOKEEPER-2621

ZooKeeper doesn't start on MINGW32 (Windows)

Bug Closed Major Fixed Amichai Rothman Amichai Rothman Amichai Rothman 26/Oct/16 05:45   20/May/19 13:51 18/Mar/19 01:27 3.4.9 3.6.0, 3.5.5, 3.4.15 scripts   0 6 0 2400   MINGW32_NT-6.1 on Windows 7 (e.g. git bash) The ZooKeeper scripts fail due to missing cygpath path conversion in a MINGW32 environment, such as when running from git bash (installed by default when installing Git for Windows).

The fix is to add the line
{quote}
MINGW*) cygwin=true ;;
{quote}
near the bottom of the zkEnv.sh script, in the case statement that checks for a cygwin environment.
100% 100% 2400 0 pull-request-available 9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
1 year, 3 days ago 0|i35eqn:
ZooKeeper ZOOKEEPER-2620

Add comments to testReadOnlySnapshotDir and testReadOnlyTxnLogDir indicating that the tests will fail when run as root

Improvement Closed Major Fixed Abraham Fine Abraham Fine Abraham Fine 24/Oct/16 17:03   31/Mar/17 05:01 05/Jan/17 16:33 3.4.9, 3.5.2 3.4.10, 3.5.3, 3.6.0 tests   0 4   testReadOnlySnapshotDir and testReadOnlyTxnLogDir test the impact of changes to file system permissions on ZooKeeper server startup.

After debugging test failures [~hanm] was experiencing, we noticed that when the unit tests are run as root, these tests fail. We should have a comment to clarify this.
9223372036854775807 No Perforce job exists for this issue. 2 9223372036854775807
3 years, 11 weeks ago 0|i35bqf:
ZooKeeper ZOOKEEPER-2619

Client library reconnecting breaks FIFO client order

Bug Open Major Unresolved Unassigned Diego Ongaro Diego Ongaro 21/Oct/16 20:01   03/Nov/16 10:54           0 9   According to the USENIX ATC 2010 [paper|https://www.usenix.org/conference/usenix-atc-10/zookeeper-wait-free-coordination-internet-scale-systems], ZooKeeper provides "FIFO client order: all requests from a given client are executed in the order that they were sent by the client." I believe applications written using the Java client library are unable to rely on this guarantee, and any current application that does so is broken. Other client libraries are also likely to be affected.

Consider this application, which is simplified from the algorithm described on Page 4 (right column) of the paper:
{code}
zk = new ZooKeeper(...)
zk.createAsync("/data-23857", "...", callback)
zk.createSync("/pointer", "/data-23857")
{code}
Assume an empty ZooKeeper database to begin with and no other writers. Applying the above definition, if the ZooKeeper database contains /pointer, it must also contain /data-23857.

Now consider this series of unfortunate events:
{code}
zk = new ZooKeeper(...)
// The library establishes a TCP connection.
zk.createAsync("/data-23857", "...", callback)
// The library/kernel closes the TCP connection because it times out, and
// the create of /data-23857 is doomed to fail with ConnectionLoss. Suppose
// that it never reaches the server.
// The library establishes a new TCP connection.
zk.createSync("/pointer", "/data-23857")
// The create of /pointer succeeds.
{code}
That's the problem: subsequent operations get assigned to the new connection and succeed, while earlier operations fail.

In general, I believe it's impossible to have a system with the following three properties:
# FIFO client order for asynchronous operations,
# Failing operations when connections are lost, AND
# Transparently reconnecting when connections are lost.

To argue this, consider an application that issues a series of pipelined operations, then upon noticing a connection loss, issues a series of recovery operations, repeating the recovery procedure as necessary. If a pipelined operation fails, all subsequent operations in the pipeline must also fail. Yet the client must also carry on eventually: the recovery operations cannot be trivially failed forever. Unfortunately, the client library does not know where the pipelined operations end and the recovery operations begin. At the time of a connection loss, subsequent pipelined operations may or may not be queued in the library; others might be upcoming in the application thread. If the library re-establishes a connection too early, it will send pipelined operations out of FIFO client order.


I considered a possible workaround of having the client diligently check its callbacks and watchers for connection loss events, and do its best to stop the subsequent pipelined operations at the first sign of a connection loss. In addition to being a large burden for the application, this does not solve the problem all the time. In particular, if the callback thread is delayed significantly (as can happen due to excessive computation or scheduling hiccups), the application may not learn about the connection loss event until after the connection has been re-established and after dependent pipelined operations have already been transmitted over the new connection.


I suggest the following API changes to fix the problem:
- Add a method ZooKeeper.getConnection() returning a ZKConnection object. ZKConnection would wrap a TCP connection. It would include all synchronous and asynchronous operations currently defined on the ZooKeeper class. Upon a connection loss on a ZKConnection, all subsequent operations on the same ZKConnection would return a Connection Loss error. Upon noticing, the client would need to call ZooKeeper.getConnection() again to get a working ZKConnection object, and it would execute its recovery procedure on this new connection.
- Deprecate all asynchronous methods on the ZooKeeper object. These are unsafe to use if the caller assumes they're getting FIFO client order.
- No changes to the protocols or servers are required.

I recognize this could cause a lot of code churn for both ZooKeeper and projects that use it. On the other hand, the existing asynchronous calls in applications should now be audited anyhow.


The code affected by this issue may be difficult to contain:
- It likely affects all ZooKeeper client libraries that provide both asynchronous operations and transparent reconnection. That's probably all versions of the official Java client library, as well as most other client libraries.
- It affects all applications using those libraries that depend on the FIFO client order of asynchronous operations. I don't know how common that is, but the paper implies that FIFO client order is important.
- Fortunately, the issue can only manifest itself when connections are lost and transparently reestablished. In practice, it may also require a long pipeline or a significant delay in the application thread while the library establishes a new connection.
- In case you're wondering, this issue occurred to me while working on a new client library for Go. I haven't seen this issue in the wild, but I was able to produce it locally by placing sleep statements in a Java program and closing its TCP connections.


I'm new to this community, so I'm looking forward to the discussion. Let me know if I can clarify any of the above.
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 20 weeks ago 0|i3591r:
ZooKeeper ZOOKEEPER-2618

ZOOKEEPER-1394 fix ClassNotFoundException on shutdown of client

Sub-task Resolved Minor Duplicate wu wen wu wen wu wen 21/Oct/16 11:11   06/Dec/16 05:18 25/Oct/16 22:24 3.4.9   java client   0 3   see ZOOKEEPER-1394,We also have this issue.

2016-10-21 13:17:21.618 ERROR localhost-startStop-1-SendThread(172.21.134.7:2005) ClientCnxn:414 - from localhost-startStop-1-SendThread(172.21.134.7:2005)
java.lang.NoClassDefFoundError: org/apache/zookeeper/server/ZooTrace
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1128)
Caused by: java.lang.ClassNotFoundException: Illegal access: this web application instance has been stopped already. Could not load [org.apache.zookeeper.server.ZooTrace]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.
at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForClassLoading(WebappClassLoaderBase.java:1315)
at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1178)
at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1139)
... 1 more
Caused by: java.lang.IllegalStateException: Illegal access: this web application instance has been stopped already. Could not load [org.apache.zookeeper.server.ZooTrace]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.
at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForResourceLoading(WebappClassLoaderBase.java:1325)
at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForClassLoading(WebappClassLoaderBase.java:1313)
... 3 more
9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
Patch
3 years, 21 weeks, 1 day ago 0|i357zj:
ZooKeeper ZOOKEEPER-2617

correct a few spelling typos

Bug Closed Trivial Fixed tony mancill tony mancill tony mancill 18/Oct/16 01:14   17/May/17 23:50 13/Feb/17 21:45 3.4.9 3.4.10, 3.5.3, 3.6.0     0 9   While working on the Debian packaging of ZooKeeper, some misspellings were detected in the source that affect the documentation, logging, and program output.

There is a PR against github containing the patch here: https://github.com/apache/zookeeper/pull/87
newbie, patch 9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
Patch
3 years, 5 weeks, 2 days ago 0|i350on:
ZooKeeper ZOOKEEPER-2616

ZK client fails to connect to ReadOnly server

Bug Open Critical Unresolved Unassigned Benjamin Jaton Benjamin Jaton 17/Oct/16 18:08   30/Jan/17 15:34   3.5.2       0 5   Only 1 of the 3 nodes of the ensemble is started. The server successfully started in readonly ("Read-only server started").

{code:title=client}System.setProperty("readonlymode.enabled", "true");

String cs = "QA-E8WIN11:2181,QA-E8WIN12:2181,QA-E8WIN13:2181";
ZooKeeper zk = new ZooKeeper(cs, 30000, null, true);

// wait for connection
while (!zk.getState().isConnected()) {
Thread.sleep(1000);
logger.error(zk.getState());
}
zk.getData("/", false, new Stat());

logger.error("DONE");{code}


The client code above manages to acquire a connection ("CONNECTEDREADONLY") but the subsequent getData fails with ConnectionLoss:

{code:title=client log}2016-10-17 14:37:43 ERROR TestCuratorReadOnly:31 - CONNECTEDREADONLY
2016-10-17 14:39:49 ERROR o.a.z.ClientCnxn:526 - Error while calling watcher
java.lang.NullPointerException
at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:524) [zookeeper-3.5.2-alpha.jar:3.5.2-alpha--1]
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:499) [zookeeper-3.5.2-alpha.jar:3.5.2-alpha--1]
Exception in thread "main" org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /
at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1956)
at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1985)
at TestCuratorReadOnly.main(TestCuratorReadOnly.java:33){code}

Full server logs are attached, but here are the relevant parts:

{code:title=server log}
2016-10-17 14:37:31,375 [myid:1] - INFO [Thread-2:ReadOnlyZooKeeperServer@73] - Read-only server started
(...)
2016-10-17 14:37:55,241 [myid:1] - INFO [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /10.11.12.4:40800
2016-10-17 14:37:55,250 [myid:1] - INFO [NIOWorkerThread-1:ZooKeeperServer@964] - Client attempting to establish new session at /10.11.12.4:40800
2016-10-17 14:37:55,255 [myid:1] - INFO [ProcessThread(sid:1 cport:-1)::ZooKeeperServer@678] - Established session 0x100024619520000 with negotiated timeout 30000 for client /10.11.12.4:40800
(...)
[org.apache.ZooKeeperService:name0=ReplicatedServer_id1,name1=replica.1,name2=ReadOnlyServer,name3=Connections,name4=10.11.12.4,name5=0x100024619520000]
2016-10-17 14:38:26,929 [myid:1] - INFO [ProcessThread(sid:1 cport:-1)::NIOServerCnxn@607] - Closed socket connection for client /10.11.12.4:40800 which had sessionid 0x100024619520000{code}

The client and server are using official 3.5.2-alpha.

{code:title=zoo.cfg}autopurge.purgeInterval=3
initLimit=10
syncLimit=5
autopurge.snapRetainCount=3
snapCount=10000
minSessionTimeout=5000
maxSessionTimeout=600000
tickTime=2000
admin.commandURL=/commands
quorumListenOnAllIPs=true
dataDir=C:/workspace/zookeeper-3.5.2-alpha/data
admin.serverPort=8080
admin.enableServer=false
standaloneEnabled=false
dynamicConfigFile=C:/workspace/zookeeper-3.5.2-alpha/conf/zoo.cfg.dynamic.10000046b{code}

{code:title=zoo.cfg.dynamic.10000046b}server.1=QA-E8WIN11:2888:3888:participant;0.0.0.0:2181
server.2=QA-E8WIN12:2888:3888:participant;0.0.0.0:2181
server.3=QA-E8WIN13:2888:3888:participant;0.0.0.0:2181{code}
9223372036854775807 No Perforce job exists for this issue. 5 9223372036854775807
3 years, 7 weeks, 3 days ago 0|i3508f:
ZooKeeper ZOOKEEPER-2615

Zookeeper server holds onto dead/expired session ids in the watch data structures

Bug Open Major Unresolved Camille Fournier guoping.gp guoping.gp 14/Oct/16 00:59   05/Feb/20 07:16   3.4.6 3.7.0, 3.5.8 server   0 7   The same issue (https://issues.apache.org/jira/browse/ZOOKEEPER-1382) still can be found even with zookeeper 3.4.6.

this issue cause our production zookeeper cluster leak about 1 million watchs, after restart the server one by one, the watch count decrease to only about 40000.

I can reproduce the issue on my mac,here it is:
------------------------------------------------------------------------

pguodeMacBook-Air:bin pguo$ echo srvr | nc localhost 6181
Zookeeper version: 3.4.6-1569965, built on 02/20/2014 09:09 GMT
Latency min/avg/max: 0/1156/128513
Received: 539
Sent: 531
Connections: 1
Outstanding: 0
Zxid: 0x100000037
Mode: follower
Node count: 5
------------------------
pguodeMacBook-Air:bin pguo$ echo cons | nc localhost 6181
/127.0.0.1:55759[1](queued=0,recved=5,sent=5,sid=0x157be2732d0000e,lop=PING,est=1476372631116,to=15000,lcxid=0x1,lzxid=0xffffffffffffffff,lresp=1476372646260,llat=8,minlat=0,avglat=6,maxlat=17)
/0:0:0:0:0:0:0:1:55767[0](queued=0,recved=1,sent=0)

------------------------
pguodeMacBook-Air:bin pguo$ echo wchp | nc localhost 6181
/curator_exists_watch
0x357be48e4d90007
0x357be48e4d90009
0x157be2732d0000e


as above 4-letter's report shows, 0x357be48e4d90007 and 0x357be48e4d90009 are leaked after the two sessions expired
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 20 weeks, 6 days ago 0|i34vnr:
ZooKeeper ZOOKEEPER-2614

Port ZOOKEEPER-1576 to branch3.4

Bug Resolved Major Fixed Thomas Schüttel Vishal Khandelwal Vishal Khandelwal 12/Oct/16 03:02   19/Aug/17 07:24 01/Aug/17 11:55 3.4.9 3.4.11     0 8   ZOOKEEPER-1576 handles UnknownHostException and it good to have this change for 3.4 branch as well. Porting the changes to 3.4 after resolving the conflicts 9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
2 years, 30 weeks, 5 days ago 0|i34rkv:
ZooKeeper ZOOKEEPER-2613

user-level KeeperException

Bug Open Major Unresolved Unassigned venkata puvvada venkata puvvada 11/Oct/16 13:31   11/Oct/16 13:31           0 1   Apache SOLR 4.10 in Zookeeper We are facing very big problem in production in Zookeeper. Its been working perfectly from many months, but suddenly without done any change in config. We had below problem. Because of the below problem, we are unable to do anything on this collection.

2016-10-11 09:54:56,448 [myid:2] - INFO [ProcessThread(sid:2 cport:-1)::PrepRequestProcessor@651] - Got user-level KeeperException when processing sessionid:0x156f39ec13e00b9 type:setData cxid:0x4728359 zxid:0xa00560776 txntype:-1 reqpath:n/a Error Path:/solr/configs/constants/managed-schema Error:KeeperErrorCode = BadVersion for /solr/configs/constants/managed-schema
^C
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 23 weeks, 2 days ago 0|i34qlj:
ZooKeeper ZOOKEEPER-2612

user-level KeeperException

Bug Open Major Unresolved Unassigned venkata puvvada venkata puvvada 11/Oct/16 13:31   11/Oct/16 13:31           0 2   Apache SOLR 4.10 in Zookeeper We are facing very big problem in production in Zookeeper. Its been working perfectly from many months, but suddenly without done any change in config. We had below problem. Because of the below problem, we are unable to do anything on this collection.

2016-10-11 09:54:56,448 [myid:2] - INFO [ProcessThread(sid:2 cport:-1)::PrepRequestProcessor@651] - Got user-level KeeperException when processing sessionid:0x156f39ec13e00b9 type:setData cxid:0x4728359 zxid:0xa00560776 txntype:-1 reqpath:n/a Error Path:/solr/configs/constants/managed-schema Error:KeeperErrorCode = BadVersion for /solr/configs/constants/managed-schema
^C
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 23 weeks, 2 days ago 0|i34qlb:
ZooKeeper ZOOKEEPER-2611

zoo_remove_watchers - can remove the wrong watch

Bug Closed Critical Fixed Eyal leshem Eyal leshem Eyal leshem 09/Oct/16 04:24   16/Dec/18 09:50 09/Oct/16 16:47   3.5.3, 3.6.0 c client   0 5   The actual problem is in the function "removeWatcherFromList" -
That when we check if we need to delete the watch - we compare the WatcherCtx to one node before the one we want to delete..
remove_watches 9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
3 years, 22 weeks, 4 days ago 0|i34mrr:
ZooKeeper ZOOKEEPER-2610

ZOOKEEPER-3170 Flaky Test: org.apache.zookeeper.test.ReconfigTest.testQuorumSystemChange

Sub-task Resolved Major Not A Problem Andor Molnar Michael Han Michael Han 08/Oct/16 14:13   19/Dec/19 17:59 25/Oct/18 10:36 3.5.2   quorum, server, tests   0 3   {noformat}
Regression

org.apache.zookeeper.test.ReconfigTest.testQuorumSystemChange (from org.apache.zookeeper.test.NioNettySuiteTest)

Failing for the past 1 build (Since Failed#3462 )
Took 2 min 10 sec.
Error Message

Waiting for server up
Stacktrace

junit.framework.AssertionFailedError: Waiting for server up
at org.apache.zookeeper.test.QuorumUtil.restart(QuorumUtil.java:216)
at org.apache.zookeeper.test.ReconfigTest.testQuorumSystemChange(ReconfigTest.java:861)
at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:79)
Standard Output

2016-10-04 01:04:03,140 [myid:] - INFO [main:JUnit4ZKTestRunner@47] - No test.method specified. using default methods.
2016-10-04 01:04:03,170 [myid:] - INFO [main:JUnit4ZKTestRunner@47] - No test.method specified. using default methods.
2016-10-04 01:04:03,179 [myid:] - INFO [main:JUnit4ZKTestRunner@47] - No test.method specified. using default methods.
2016-10-04 01:04:03,182 [myid:] - INFO [main:JUnit4ZKTestRunner@47] - No test.method specified. using default methods.
2016-10-04 01:04:03,186 [myid:] - INFO [main:JUnit4ZKTestRunner@47] - No test.method specified. using default methods.
2016-10-04 01:04:03,190 [myid:] - INFO [main:JUnit4ZKTestRunner@47] - No test.method specified. using default methods.
2016-10-04 01:04:03,200 [myid:] - INFO [main:JUnit4ZKTestRunner@47] - No test.method specified. using default methods.
2016-10-04 01:04:03,204 [myid:] - INFO [main:JUnit4ZKTestRunner@47] - No test.method specified. using default methods.
2016-10-04 01:04:03,208 [myid:] - INFO [main:JUnit4ZKTestRunner@47] - No test.method specified. using default methods.
2016-10-04 01:04:03,276 [myid:] - INFO [main:JUnit4ZKTestRunner@47] - No test.method specified. using default methods.
2016-10-04 01:04:03,279 [myid:] - INFO [main:JUnit4ZKTestRunner@47] - No test.method specified. using default methods.
2016-10-04 01:04:03,282 [myid:] - INFO [main:JUnit4ZKTestRunner@47] - No test.method specified. using default methods.
2016-10-04 01:04:03,284 [myid:] - INFO [main:JUnit4ZKTestRunner@47] - No test.method specified. using default methods.
2016-10-04 01:04:03,285 [myid:] - INFO [main:JUnit4ZKTestRunner@47] - No test.method specified. using default methods.
2016-10-04 01:04:03,286 [myid:] - INFO [main:JUnit4ZKTestRunner@47] - No test.method specified. using default methods.
2016-10-04 01:04:03,286 [myid:] - INFO [main:JUnit4ZKTestRunner@47] - No test.method specified. using default methods.
2016-10-04 01:04:03,291 [myid:] - INFO [main:JUnit4ZKTestRunner@47] - No test.method specified. using default methods.
2016-10-04 01:04:03,292 [myid:] - INFO [main:JUnit4ZKTestRunner@47] - No test.method specified. using default methods.
2016-10-04 01:04:03,304 [myid:] - INFO [main:PortAssignment@151] - Test process 3/8 using ports from 16607 - 19299.
2016-10-04 01:04:03,308 [myid:] - INFO [main:PortAssignment@85] - Assigned port 16608 from range 16607 - 19299.
2016-10-04 01:04:03,316 [myid:] - INFO [main:ZKTestCase$1@55] - STARTING testAcls
2016-10-04 01:04:03,317 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@77] - RUNNING TEST METHOD testAcls
2016-10-04 01:04:03,337 [myid:] - INFO [main:Environment@109] - Server environment:zookeeper.version=3.6.0-SNAPSHOT--1, built on 10/04/2016 00:56 GMT
2016-10-04 01:04:03,337 [myid:] - INFO [main:Environment@109] - Server environment:host.name=asf905.gq1.ygridcore.net
2016-10-04 01:04:03,337 [myid:] - INFO [main:Environment@109] - Server environment:java.version=1.7.0_80
2016-10-04 01:04:03,337 [myid:] - INFO [main:Environment@109] - Server environment:java.vendor=Oracle Corporation
2016-10-04 01:04:03,337 [myid:] - INFO [main:Environment@109] - Server environment:java.home=/usr/local/asfpackages/java/jdk1.7.0_80/jre
2016-10-04 01:04:03,337 [myid:] - INFO [main:Environment@109] - Server environment:java.class.path=/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/build/test/classes:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/build/test/lib/antlr-2.7.7.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/build/test/lib/antlr4-runtime-4.5.1-1.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/build/test/lib/checkstyle-6.13.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/build/test/lib/commons-beanutils-1.9.2.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/build/test/lib/commons-cli-1.3.1.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/build/test/lib/commons-collections-3.2.2.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/build/test/lib/commons-lang3-3.4.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/build/test/lib/commons-logging-1.1.1.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/build/test/lib/guava-18.0.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/build/test/lib/hamcrest-core-1.3.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/build/test/lib/junit-4.12.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/build/test/lib/mockito-all-1.8.2.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/build/classes:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/src/java/lib/ivy-2.4.0.jar:/home/jenkins/tools/ant/latest/lib/ant.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/build/lib/apache-rat-core-0.10.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/build/lib/apache-rat-tasks-0.10.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/build/lib/commons-cli-1.2.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/build/lib/commons-collections-3.2.2.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/build/lib/commons-compress-1.5.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/build/lib/commons-io-2.2.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/build/lib/commons-lang-2.6.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/build/lib/jackson-core-asl-1.9.11.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/build/lib/jackson-mapper-asl-1.9.11.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/build/lib/javacc.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/build/lib/javax.servlet-api-3.1.0.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/build/lib/jetty-http-9.2.18.v20160721.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/build/lib/jetty-io-9.2.18.v20160721.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/build/lib/jetty-security-9.2.18.v20160721.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/build/lib/jetty-server-9.2.18.v20160721.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/build/lib/jetty-servlet-9.2.18.v20160721.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/build/lib/jetty-util-9.2.18.v20160721.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/build/lib/jline-2.11.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/build/lib/log4j-1.2.17.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/build/lib/netty-3.10.5.Final.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/build/lib/slf4j-api-1.7.5.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/build/lib/slf4j-log4j12-1.7.5.jar:/usr/local/asfpackages/ant/apache-ant-1.9.7/lib/ant-launcher.jar:/home/jenkins/tools/ant/latest/lib/ant-junit.jar:/home/jenkins/tools/ant/latest/lib/ant-junit4.jar
2016-10-04 01:04:03,338 [myid:] - INFO [main:Environment@109] - Server environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2016-10-04 01:04:03,338 [myid:] - INFO [main:Environment@109] - Server environment:java.io.tmpdir=/tmp
2016-10-04 01:04:03,338 [myid:] - INFO [main:Environment@109] - Server environment:java.compiler=<NA>
2016-10-04 01:04:03,338 [myid:] - INFO [main:Environment@109] - Server environment:os.name=Linux
2016-10-04 01:04:03,339 [myid:] - INFO [main:Environment@109] - Server environment:os.arch=amd64
2016-10-04 01:04:03,339 [myid:] - INFO [main:Environment@109] - Server environment:os.version=3.13.0-95-generic
2016-10-04 01:04:03,339 [myid:] - INFO [main:Environment@109] - Server environment:user.name=jenkins
2016-10-04 01:04:03,339 [myid:] - INFO [main:Environment@109] - Server environment:user.home=/home/jenkins
2016-10-04 01:04:03,339 [myid:] - INFO [main:Environment@109] - Server environment:user.dir=/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/build/test
2016-10-04 01:04:03,339 [myid:] - INFO [main:Environment@109] - Server environment:os.memory.free=475MB
2016-10-04 01:04:03,340 [myid:] - INFO [main:Environment@109] - Server environment:os.memory.max=491MB
2016-10-04 01:04:03,340 [myid:] - INFO [main:Environment@109] - Server environment:os.memory.total=491MB
2016-10-04 01:04:03,357 [myid:] - INFO [main:ZooKeeperServer@889] - minSessionTimeout set to 6000
2016-10-04 01:04:03,357 [myid:] - INFO [main:ZooKeeperServer@898] - maxSessionTimeout set to 60000
2016-10-04 01:04:03,357 [myid:] - INFO [main:ZooKeeperServer@159] - Created server with tickTime 3000 minSessionTimeout 6000 maxSessionTimeout 60000 datadir /home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/build/test/tmp/test1469455732852767377.junit.dir/version-2 snapdir /home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/build/test/tmp/test1469455732852767377.junit.dir/version-2
2016-10-04 01:04:03,469 [myid:] - INFO [main:NettyServerCnxnFactory@487] - binding to port 0.0.0.0/0.0.0.0:16608
2016-10-04 01:04:03,509 [myid:] - INFO [main:FileTxnSnapLog@306] - Snapshotting: 0x0 to /home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/build/test/tmp/test1469455732852767377.junit.dir/version-2/snapshot.0
2016-10-04 01:04:03,614 [myid:] - ERROR [main:ZooKeeperServer@501] - ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
2016-10-04 01:04:03,615 [myid:] - INFO [main:ACLTest@108] - starting up the zookeeper server .. waiting
2016-10-04 01:04:03,617 [myid:] - INFO [main:FourLetterWordMain@85] - connecting to 127.0.0.1 16608
2016-10-04 01:04:03,652 [myid:] - INFO [New I/O worker #1:NettyServerCnxn@275] - Processing stat command from /127.0.0.1:52750
2016-10-04 01:04:03,660 [myid:] - INFO [New I/O worker #1:StatCommand@49] - Stat command output
2016-10-04 01:04:03,670 [myid:] - INFO [main:Environment@109] - Client environment:zookeeper.version=3.6.0-SNAPSHOT--1, built on 10/04/2016 00:56 GMT
2016-10-04 01:04:03,671 [myid:] - INFO [main:Environment@109] - Client environment:host.name=asf905.gq1.ygridcore.net
2016-10-04 01:04:03,671 [myid:] - INFO [main:Environment@109] - Client environment:java.version=1.7.0_80
2016-10-04 01:04:03,671 [myid:] - INFO [main:Environment@109] - Client environment:java.vendor=Oracle Corporation
2016-10-04 01:04:03,671 [myid:] - INFO [main:Environment@109] - Client environment:java.home=/usr/local/asfpackages/java/jdk1.7.0_80/jre
2016-10-04 01:04:03,671 [myid:] - INFO [main:Environment@109] - Client environment:java.class.path=/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/build/test/classes:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/build/test/lib/antlr-2.7.7.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/build/test/lib/antlr4-runtime-4.5.1-1.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/build/test/lib/checkstyle-6.13.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/build/test/lib/commons-beanutils-1.9.2.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/build/test/lib/commons-cli-1.3.1.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/build/test/lib/commons-collections-3.2.2.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/build/test/lib/commons-lang3-3.4.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/build/test/lib/commons-logging-1.1.1.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/build/test/lib/guava-18.0.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/build/test/lib/hamcrest-core-1.3.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/build/test/lib/junit-4.12.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/build/test/lib/mockito-all-1.8.2.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/build/classes:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/src/java/lib/ivy-2.4.0.jar:/home/jenkins/tools/ant/latest/lib/ant.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/build/lib/apache-rat-core-0.10.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/build/lib/apache-rat-tasks-0.10.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/build/lib/commons-cli-1.2.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/build/lib/commons-collections-3.2.2.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/build/lib/commons-compress-1.5.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/build/lib/commons-io-2.2.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/build/lib/commons-lang-2.6.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/build/lib/jackson-core-asl-1.9.11.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/build/lib/jackson-mapper-asl-1.9.11.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/build/lib/javacc.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/build/lib/javax.servlet-api-3.1.0.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/build/lib/jetty-http-9.2.18.v20160721.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/build/lib/jetty-io-9.2.18.v20160721.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/build/lib/jetty-security-9.2.18.v20160721.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/build/lib/jetty-server-9.2.18.v20160721.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/build/lib/jetty-servlet-9.2.18.v20160721.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/build/lib/jetty-util-9.2.18.v20160721.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/build/lib/jline-2.11.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/build/lib/log4j-1.2.17.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/build/lib/netty-3.10.5.Final.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/build/lib/slf4j-api-1.7.5.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/build/lib/slf4j-log4j12-1.7.5.jar:/usr/local/asfpackages/ant/apache-ant-1.9.7/lib/ant-launcher.jar:/home/jenkins/tools/ant/latest/lib/ant-junit.jar:/home/jenkins/tools/ant/latest/lib/ant-junit4.jar
2016-10-04 01:04:03,672 [myid:] - INFO [main:Environment@109] - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2016-10-04 01:04:03,672 [myid:] - INFO [main:Environment@109] - Client environment:java.io.tmpdir=/tmp
2016-10-04 01:04:03,672 [myid:] - INFO [main:Environment@109] - Client environment:java.compiler=<NA>
2016-10-04 01:04:03,672 [myid:] - INFO [main:Environment@109] - Client environment:os.name=Linux
2016-10-04 01:04:03,672 [myid:] - INFO [main:Environment@109] - Client environment:os.arch=amd64
2016-10-04 01:04:03,672 [myid:] - INFO [main:Environment@109] - Client environment:os.version=3.13.0-95-generic
2016-10-04 01:04:03,673 [myid:] - INFO [main:Environment@109] - Client environment:user.name=jenkins
2016-10-04 01:04:03,673 [myid:] - INFO [main:Environment@109] - Client environment:user.home=/home/jenkins
2016-10-04 01:04:03,673 [myid:] - INFO [main:Environment@109] - Client environment:user.dir=/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/build/test
2016-10-04 01:04:03,673 [myid:] - INFO [main:Environment@109] - Client environment:os.memory.free=377MB
2016-10-04 01:04:03,673 [myid:] - INFO [main:Environment@109] - Client environment:os.memory.max=491MB
2016-10-04 01:04:03,674 [myid:] - INFO [main:Environment@109] - Client environment:os.memory.total=491MB
2016-10-04 01:04:03,676 [myid:] - INFO [main:ZooKeeper@853] - Initiating client connection, connectString=127.0.0.1:16608 sessionTimeout=30000 watcher=org.apache.zookeeper.test.ClientBase$CountdownWatcher@40ae49d5
2016-10-04 01:04:03,693 [myid:127.0.0.1:16608] - INFO [main-SendThread(127.0.0.1:16608):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:16608. Will not attempt to authenticate using SASL (unknown error)
2016-10-04 01:04:03,694 [myid:127.0.0.1:16608] - INFO [main-SendThread(127.0.0.1:16608):ClientCnxn$SendThread@948] - Socket connection established, initiating session, client: /127.0.0.1:52752, server: 127.0.0.1/127.0.0.1:16608
2016-10-04 01:04:03,698 [myid:] - INFO [New I/O worker #2:ZooKeeperServer@995] - Client attempting to establish new session at /127.0.0.1:52752
2016-10-04 01:04:03,702 [myid:] - INFO [SyncThread:0:FileTxnLog@204] - Creating new log file: log.1
2016-10-04 01:04:03,782 [myid:] - INFO [SyncThread:0:ZooKeeperServer@709] - Established session 0x10020f0235b0000 with negotiated timeout 30000 for client /127.0.0.1:52752
2016-10-04 01:04:03,782 [myid:127.0.0.1:16608] - INFO [main-SendThread(127.0.0.1:16608):ClientCnxn$SendThread@1381] - Session establishment complete on server 127.0.0.1/127.0.0.1:16608, sessionid = 0x10020f0235b0000, negotiated timeout = 30000
2016-10-04 01:04:03,785 [myid:] - INFO [main:ACLTest@112] - starting creating acls
2016-10-04 01:04:08,518 [myid:] - INFO [main:NettyServerCnxnFactory@464] - shutdown called 0.0.0.0/0.0.0.0:16608
2016-10-04 01:04:08,519 [myid:127.0.0.1:16608] - INFO [main-SendThread(127.0.0.1:16608):ClientCnxn$SendThread@1231] - Unable to read additional data from server sessionid 0x10020f0235b0000, likely server has closed socket, closing socket connection and attempting reconnect
2016-10-04 01:04:08,521 [myid:] - INFO [main:MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=StandaloneServer_port16608,name1=Connections,name2=127.0.0.1,name3=0x10020f0235b0000]
2016-10-04 01:04:08,536 [myid:] - INFO [main:ZooKeeperServer@529] - shutting down
2016-10-04 01:04:08,536 [myid:] - ERROR [main:ZooKeeperServer@501] - ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
2016-10-04 01:04:08,536 [myid:] - INFO [main:SessionTrackerImpl@232] - Shutting down
2016-10-04 01:04:08,536 [myid:] - INFO [main:PrepRequestProcessor@975] - Shutting down
2016-10-04 01:04:08,537 [myid:] - INFO [main:SyncRequestProcessor@191] - Shutting down
2016-10-04 01:04:08,537 [myid:] - INFO [ProcessThread(sid:0 cport:16608)::PrepRequestProcessor@154] - PrepRequestProcessor exited loop!
2016-10-04 01:04:08,537 [myid:] - INFO [SyncThread:0:SyncRequestProcessor@169] - SyncRequestProcessor exited!
2016-10-04 01:04:08,537 [myid:] - INFO [main:FinalRequestProcessor@479] - shutdown of request processor complete
2016-10-04 01:04:08,538 [myid:] - INFO [main:MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=StandaloneServer_port16608,name1=InMemoryDataTree]
2016-10-04 01:04:08,538 [myid:] - INFO [main:MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=StandaloneServer_port16608]
2016-10-04 01:04:08,538 [myid:] - INFO [main:FourLetterWordMain@85] - connecting to 127.0.0.1 16608
2016-10-04 01:04:08,539 [myid:] - INFO [main:ZooKeeperServer@889] - minSessionTimeout set to 6000
2016-10-04 01:04:08,539 [myid:] - INFO [main:ZooKeeperServer@898] - maxSessionTimeout set to 60000
2016-10-04 01:04:08,540 [myid:] - INFO [main:ZooKeeperServer@159] - Created server with tickTime 3000 minSessionTimeout 6000 maxSessionTimeout 60000 datadir /home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/build/test/tmp/test1469455732852767377.junit.dir/version-2 snapdir /home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/build/test/tmp/test1469455732852767377.junit.dir/version-2
2016-10-04 01:04:08,565 [myid:] - INFO [main:NettyServerCnxnFactory@487] - binding to port 0.0.0.0/0.0.0.0:16608
2016-10-04 01:04:08,567 [myid:] - INFO [main:FileSnap@83] - Reading snapshot /home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/build/test/tmp/test1469455732852767377.junit.dir/version-2/snapshot.0
2016-10-04 01:04:08,598 [myid:] - INFO [main:FileTxnSnapLog@306] - Snapshotting: 0xc9 to /home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/build/test/tmp/test1469455732852767377.junit.dir/version-2/snapshot.c9
2016-10-04 01:04:08,609 [myid:] - ERROR [main:ZooKeeperServer@501] - ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
2016-10-04 01:04:08,609 [myid:] - INFO [main:FourLetterWordMain@85] - connecting to 127.0.0.1 16608
2016-10-04 01:04:08,610 [myid:] - INFO [New I/O worker #34:NettyServerCnxn@275] - Processing stat command from /127.0.0.1:52862
2016-10-04 01:04:08,610 [myid:] - INFO [New I/O worker #34:StatCommand@49] - Stat command output
2016-10-04 01:04:08,611 [myid:] - INFO [main:ZooKeeper@853] - Initiating client connection, connectString=127.0.0.1:16608 sessionTimeout=30000 watcher=org.apache.zookeeper.test.ClientBase$CountdownWatcher@19450f1a
2016-10-04 01:04:08,612 [myid:127.0.0.1:16608] - INFO [main-SendThread(127.0.0.1:16608):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:16608. Will not attempt to authenticate using SASL (unknown error)
2016-10-04 01:04:08,613 [myid:127.0.0.1:16608] - INFO [main-SendThread(127.0.0.1:16608):ClientCnxn$SendThread@948] - Socket connection established, initiating session, client: /127.0.0.1:52863, server: 127.0.0.1/127.0.0.1:16608
2016-10-04 01:04:08,614 [myid:] - INFO [New I/O worker #35:ZooKeeperServer@995] - Client attempting to establish new session at /127.0.0.1:52863
2016-10-04 01:04:08,615 [myid:] - INFO [SyncThread:0:FileTxnLog@204] - Creating new log file: log.ca
2016-10-04 01:04:08,637 [myid:] - INFO [SyncThread:0:ZooKeeperServer@709] - Established session 0x10020f0373e0000 with negotiated timeout 30000 for client /127.0.0.1:52863
2016-10-04 01:04:08,638 [myid:127.0.0.1:16608] - INFO [main-SendThread(127.0.0.1:16608):ClientCnxn$SendThread@1381] - Session establishment complete on server 127.0.0.1/127.0.0.1:16608, sessionid = 0x10020f0373e0000, negotiated timeout = 30000
2016-10-04 01:04:08,681 [myid:] - INFO [ProcessThread(sid:0 cport:16608)::PrepRequestProcessor@657] - Processed session termination for sessionid: 0x10020f0373e0000
2016-10-04 01:04:08,690 [myid:] - INFO [main:ZooKeeper@1311] - Session: 0x10020f0373e0000 closed
2016-10-04 01:04:08,690 [myid:] - INFO [SyncThread:0:MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=StandaloneServer_port16608,name1=Connections,name2=127.0.0.1,name3=0x10020f0373e0000]
2016-10-04 01:04:08,690 [myid:] - INFO [main-EventThread:ClientCnxn$EventThread@513] - EventThread shut down for session: 0x10020f0373e0000
2016-10-04 01:04:08,691 [myid:] - INFO [main:NettyServerCnxnFactory@464] - shutdown called 0.0.0.0/0.0.0.0:16608
2016-10-04 01:04:08,717 [myid:] - INFO [main:ZooKeeperServer@529] - shutting down
2016-10-04 01:04:08,717 [myid:] - ERROR [main:ZooKeeperServer@501] - ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
2016-10-04 01:04:08,717 [myid:] - INFO [main:SessionTrackerImpl@232] - Shutting down
2016-10-04 01:04:08,717 [myid:] - INFO [main:PrepRequestProcessor@975] - Shutting down
2016-10-04 01:04:08,718 [myid:] - INFO [main:SyncRequestProcessor@191] - Shutting down
2016-10-04 01:04:08,718 [myid:] - INFO [ProcessThread(sid:0 cport:16608)::PrepRequestProcessor@154] - PrepRequestProcessor exited loop!
2016-10-04 01:04:08,718 [myid:] - INFO [SyncThread:0:SyncRequestProcessor@169] - SyncRequestProcessor exited!
2016-10-04 01:04:08,718 [myid:] - INFO [main:FinalRequestProcessor@479] - shutdown of request processor complete
2016-10-04 01:04:08,719 [myid:] - INFO [main:MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=StandaloneServer_port16608,name1=InMemoryDataTree]
2016-10-04 01:04:08,719 [myid:] - INFO [main:MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=StandaloneServer_port16608]
2016-10-04 01:04:08,720 [myid:] - INFO [main:FourLetterWordMain@85] - connecting to 127.0.0.1 16608
2016-10-04 01:04:08,721 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@82] - Memory used 39623
2016-10-04 01:04:08,721 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@87] - Number of threads 9
2016-10-04 01:04:08,721 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@102] - FINISHED TEST METHOD testAcls
2016-10-04 01:04:08,721 [myid:] - INFO [main:ZKTestCase$1@65] - SUCCEEDED testAcls
2016-10-04 01:04:08,722 [myid:] - INFO [main:ZKTestCase$1@60] - FINISHED testAcls
2016-10-04 01:04:08,723 [myid:] - INFO [main:ZKTestCase$1@55] - STARTING testIPAuthenticationIsValidCIDR
2016-10-04 01:04:08,723 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@77] - RUNNING TEST METHOD testIPAuthenticationIsValidCIDR
2016-10-04 01:04:08,724 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@82] - Memory used 39623
2016-10-04 01:04:08,724 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@87] - Number of threads 9
2016-10-04 01:04:08,724 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@102] - FINISHED TEST METHOD testIPAuthenticationIsValidCIDR
2016-10-04 01:04:08,725 [myid:] - INFO [main:ZKTestCase$1@65] - SUCCEEDED testIPAuthenticationIsValidCIDR
2016-10-04 01:04:08,725 [myid:] - INFO [main:ZKTestCase$1@60] - FINISHED testIPAuthenticationIsValidCIDR
2016-10-04 01:04:08,741 [myid:] - INFO [main:ZKTestCase$1@55] - STARTING testDisconnectedAddAuth
2016-10-04 01:04:08,741 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@77] - RUNNING TEST METHOD testDisconnectedAddAuth
2016-10-04 01:04:08,742 [myid:] - INFO [main:ZooKeeperServer@889] - minSessionTimeout set to 6000
2016-10-04 01:04:08,742 [myid:] - INFO [main:ZooKeeperServer@898] - maxSessionTimeout set to 60000
2016-10-04 01:04:08,743 [myid:] - INFO [main:ZooKeeperServer@159] - Created server with tickTime 3000 minSessionTimeout 6000 maxSessionTimeout 60000 datadir /home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/build/test/tmp/test6791388306054347439.junit.dir/version-2 snapdir /home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/build/test/tmp/test6791388306054347439.junit.dir/version-2
2016-10-04 01:04:08,751 [myid:] - INFO [main:NettyServerCnxnFactory@487] - binding to port 0.0.0.0/0.0.0.0:16608
2016-10-04 01:04:08,753 [myid:] - INFO [main:FileTxnSnapLog@306] - Snapshotting: 0x0 to /home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/build/test/tmp/test6791388306054347439.junit.dir/version-2/snapshot.0
2016-10-04 01:04:08,755 [myid:] - ERROR [main:ZooKeeperServer@501] - ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
2016-10-04 01:04:08,755 [myid:] - INFO [main:ACLTest@72] - starting up the zookeeper server .. waiting
2016-10-04 01:04:08,755 [myid:] - INFO [main:FourLetterWordMain@85] - connecting to 127.0.0.1 16608
2016-10-04 01:04:08,757 [myid:] - INFO [New I/O worker #67:NettyServerCnxn@275] - Processing stat command from /127.0.0.1:52865
2016-10-04 01:04:08,757 [myid:] - INFO [New I/O worker #67:StatCommand@49] - Stat command output
2016-10-04 01:04:08,758 [myid:] - INFO [main:ZooKeeper@853] - Initiating client connection, connectString=127.0.0.1:16608 sessionTimeout=30000 watcher=org.apache.zookeeper.test.ClientBase$CountdownWatcher@68098843
2016-10-04 01:04:08,759 [myid:127.0.0.1:16608] - INFO [main-SendThread(127.0.0.1:16608):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:16608. Will not attempt to authenticate using SASL (unknown error)
2016-10-04 01:04:08,759 [myid:127.0.0.1:16608] - INFO [main-SendThread(127.0.0.1:16608):ClientCnxn$SendThread@948] - Socket connection established, initiating session, client: /127.0.0.1:52866, server: 127.0.0.1/127.0.0.1:16608
2016-10-04 01:04:08,760 [myid:] - INFO [New I/O worker #68:ZooKeeperServer@995] - Client attempting to establish new session at /127.0.0.1:52866
2016-10-04 01:04:08,761 [myid:] - INFO [SyncThread:0:FileTxnLog@204] - Creating new log file: log.1
2016-10-04 01:04:08,800 [myid:] - INFO [SyncThread:0:ZooKeeperServer@709] - Established session 0x10020f037d10000 with negotiated timeout 30000 for client /127.0.0.1:52866
2016-10-04 01:04:08,801 [myid:127.0.0.1:16608] - INFO [main-SendThread(127.0.0.1:16608):ClientCnxn$SendThread@1381] - Session establishment complete on server 127.0.0.1/127.0.0.1:16608, sessionid = 0x10020f037d10000, negotiated timeout = 30000
2016-10-04 01:04:08,803 [myid:] - INFO [New I/O worker #68:ZooKeeperServer@1032] - got auth packet /127.0.0.1:52866
2016-10-04 01:04:08,803 [myid:] - INFO [New I/O worker #68:ZooKeeperServer@1050] - auth success /127.0.0.1:52866
2016-10-04 01:04:08,822 [myid:] - INFO [ProcessThread(sid:0 cport:16608)::PrepRequestProcessor@657] - Processed session termination for sessionid: 0x10020f037d10000
2016-10-04 01:04:08,830 [myid:] - INFO [SyncThread:0:MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=StandaloneServer_port16608,name1=Connections,name2=127.0.0.1,name3=0x10020f037d10000]
2016-10-04 01:04:08,830 [myid:] - INFO [main:ZooKeeper@1311] - Session: 0x10020f037d10000 closed
2016-10-04 01:04:08,830 [myid:] - INFO [main-EventThread:ClientCnxn$EventThread@513] - EventThread shut down for session: 0x10020f037d10000
2016-10-04 01:04:08,830 [myid:] - INFO [main:NettyServerCnxnFactory@464] - shutdown called 0.0.0.0/0.0.0.0:16608
2016-10-04 01:04:08,839 [myid:] - INFO [main:ZooKeeperServer@529] - shutting down
2016-10-04 01:04:08,839 [myid:] - ERROR [main:ZooKeeperServer@501] - ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
2016-10-04 01:04:08,840 [myid:] - INFO [main:SessionTrackerImpl@232] - Shutting down
2016-10-04 01:04:08,841 [myid:] - INFO [main:PrepRequestProcessor@975] - Shutting down
2016-10-04 01:04:08,841 [myid:] - INFO [main:SyncRequestProcessor@191] - Shutting down
2016-10-04 01:04:08,841 [myid:] - INFO [ProcessThread(sid:0 cport:16608)::PrepRequestProcessor@154] - PrepRequestProcessor exited loop!
2016-10-04 01:04:08,843 [myid:] - INFO [SyncThread:0:SyncRequestProcessor@169] - SyncRequestProcessor exited!
2016-10-04 01:04:08,844 [myid:] - INFO [main:FinalRequestProcessor@479] - shutdown of request processor complete
2016-10-04 01:04:08,844 [myid:] - INFO [main:MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=StandaloneServer_port16608,name1=InMemoryDataTree]
2016-10-04 01:04:08,844 [myid:] - INFO [main:MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=StandaloneServer_port16608]
2016-10-04 01:04:08,845 [myid:] - INFO [main:FourLetterWordMain@85] - connecting to 127.0.0.1 16608
2016-10-04 01:04:08,845 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@82] - Memory used 67299
2016-10-04 01:04:08,845 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@87] - Number of threads 10
2016-10-04 01:04:08,846 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@102] - FINISHED TEST METHOD testDisconnectedAddAuth
2016-10-04 01:04:08,846 [myid:] - INFO [main:ZKTestCase$1@65] - SUCCEEDED testDisconnectedAddAuth
2016-10-04 01:04:08,846 [myid:] - INFO [main:ZKTestCase$1@60] - FINISHED testDisconnectedAddAuth
2016-10-04 01:04:08,847 [myid:] - INFO [main:PortAssignment@85] - Assigned port 16609 from range 16607 - 19299.
2016-10-04 01:04:08,847 [myid:] - INFO [main:ZKTestCase$1@55] - STARTING testAsyncMulti
2016-10-04 01:04:08,848 [myid:] - INFO [main:ClientBase@448] - Initial fdcount is: 60
2016-10-04 01:04:08,995 [myid:] - INFO [main:ClientBase@466] - STARTING server
2016-10-04 01:04:08,996 [myid:] - INFO [main:ClientBase@386] - CREATING server instance 127.0.0.1:16609
2016-10-04 01:04:09,004 [myid:] - INFO [main:ClientBase@361] - STARTING server instance 127.0.0.1:16609
2016-10-04 01:04:09,005 [myid:] - INFO [main:ZooKeeperServer@889] - minSessionTimeout set to 6000
2016-10-04 01:04:09,005 [myid:] - INFO [main:ZooKeeperServer@898] - maxSessionTimeout set to 60000
2016-10-04 01:04:09,005 [myid:] - INFO [main:ZooKeeperServer@159] - Created server with tickTime 3000 minSessionTimeout 6000 maxSessionTimeout 60000 datadir /home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/build/test/tmp/test484349696560279448.junit.dir/version-2 snapdir /home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/build/test/tmp/test484349696560279448.junit.dir/version-2
2016-10-04 01:04:09,006 [myid:] - INFO [main:NettyServerCnxnFactory@487] - binding to port 0.0.0.0/0.0.0.0:16609
2016-10-04 01:04:09,007 [myid:] - INFO [main:FileTxnSnapLog@306] - Snapshotting: 0x0 to /home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/build/test/tmp/test484349696560279448.junit.dir/version-2/snapshot.0
2016-10-04 01:04:09,012 [myid:] - ERROR [main:ZooKeeperServer@501] - ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
2016-10-04 01:04:09,012 [myid:] - INFO [main:FourLetterWordMain@85] - connecting to 127.0.0.1 16609
2016-10-04 01:04:09,013 [myid:] - INFO [New I/O worker #100:NettyServerCnxn@275] - Processing stat command from /127.0.0.1:49194
2016-10-04 01:04:09,014 [myid:] - INFO [New I/O worker #100:StatCommand@49] - Stat command output
2016-10-04 01:04:09,014 [myid:] - INFO [main:JMXEnv@228] - ensureParent:[InMemoryDataTree, StandaloneServer_port]
2016-10-04 01:04:09,021 [myid:] - INFO [main:JMXEnv@245] - expect:InMemoryDataTree
2016-10-04 01:04:09,021 [myid:] - INFO [main:JMXEnv@249] - found:InMemoryDataTree org.apache.ZooKeeperService:name0=StandaloneServer_port16609,name1=InMemoryDataTree
2016-10-04 01:04:09,022 [myid:] - INFO [main:JMXEnv@245] - expect:StandaloneServer_port
2016-10-04 01:04:09,022 [myid:] - INFO [main:JMXEnv@249] - found:StandaloneServer_port org.apache.ZooKeeperService:name0=StandaloneServer_port16609
2016-10-04 01:04:09,022 [myid:] - INFO [main:ClientBase@462] - Client test setup finished
2016-10-04 01:04:09,022 [myid:] - INFO [main:AsyncOpsTest@50] - Creating client testAsyncMulti
2016-10-04 01:04:09,023 [myid:] - INFO [main:ZooKeeper@853] - Initiating client connection, connectString=127.0.0.1:16609 sessionTimeout=30000 watcher=org.apache.zookeeper.test.ClientBase$CountdownWatcher@5a124d9f
2016-10-04 01:04:09,024 [myid:127.0.0.1:16609] - INFO [main-SendThread(127.0.0.1:16609):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:16609. Will not attempt to authenticate using SASL (unknown error)
2016-10-04 01:04:09,024 [myid:127.0.0.1:16609] - INFO [main-SendThread(127.0.0.1:16609):ClientCnxn$SendThread@948] - Socket connection established, initiating session, client: /127.0.0.1:49195, server: 127.0.0.1/127.0.0.1:16609
2016-10-04 01:04:09,026 [myid:] - INFO [New I/O worker #101:ZooKeeperServer@995] - Client attempting to establish new session at /127.0.0.1:49195
2016-10-04 01:04:09,026 [myid:] - INFO [SyncThread:0:FileTxnLog@204] - Creating new log file: log.1
2016-10-04 01:04:09,070 [myid:] - INFO [SyncThread:0:ZooKeeperServer@709] - Established session 0x10020f038cf0000 with negotiated timeout 30000 for client /127.0.0.1:49195
2016-10-04 01:04:09,070 [myid:127.0.0.1:16609] - INFO [main-SendThread(127.0.0.1:16609):ClientCnxn$SendThread@1381] - Session establishment complete on server 127.0.0.1/127.0.0.1:16609, sessionid = 0x10020f038cf0000, negotiated timeout = 30000
2016-10-04 01:04:09,073 [myid:] - INFO [main:JMXEnv@117] - expect:0x10020f038cf0000
2016-10-04 01:04:09,073 [myid:] - INFO [main:JMXEnv@120] - found:0x10020f038cf0000 org.apache.ZooKeeperService:name0=StandaloneServer_port16609,name1=Connections,name2=127.0.0.1,name3=0x10020f038cf0000
2016-10-04 01:04:09,073 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@77] - RUNNING TEST METHOD testAsyncMulti
2016-10-04 01:04:09,074 [myid:] - INFO [New I/O worker #101:ZooKeeperServer@1032] - got auth packet /127.0.0.1:49195
2016-10-04 01:04:09,074 [myid:] - INFO [New I/O worker #101:ZooKeeperServer@1050] - auth success /127.0.0.1:49195
2016-10-04 01:04:09,102 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@82] - Memory used 33655
2016-10-04 01:04:09,102 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@87] - Number of threads 55
2016-10-04 01:04:09,102 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@102] - FINISHED TEST METHOD testAsyncMulti
2016-10-04 01:04:09,103 [myid:] - INFO [ProcessThread(sid:0 cport:16609)::PrepRequestProcessor@657] - Processed session termination for sessionid: 0x10020f038cf0000
2016-10-04 01:04:09,116 [myid:] - INFO [SyncThread:0:MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=StandaloneServer_port16609,name1=Connections,name2=127.0.0.1,name3=0x10020f038cf0000]
2016-10-04 01:04:09,118 [myid:] - WARN [New I/O worker #101:NettyServerCnxnFactory$CnxnChannelHandler@142] - Exception caught [id: 0xd523ab2f, /127.0.0.1:49195 :> /127.0.0.1:16609] EXCEPTION: java.nio.channels.ClosedChannelException
java.nio.channels.ClosedChannelException
at sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:270)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:479)
at org.jboss.netty.channel.socket.nio.SocketSendBufferPool$UnpooledSendBuffer.transferTo(SocketSendBufferPool.java:203)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:201)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromTaskLoop(AbstractNioWorker.java:151)
at org.jboss.netty.channel.socket.nio.AbstractNioChannel$WriteTask.run(AbstractNioChannel.java:315)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:391)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:315)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2016-10-04 01:04:09,216 [myid:] - INFO [main:ZooKeeper@1311] - Session: 0x10020f038cf0000 closed
2016-10-04 01:04:09,217 [myid:] - INFO [main:ClientBase@543] - tearDown starting
2016-10-04 01:04:09,217 [myid:] - INFO [main-EventThread:ClientCnxn$EventThread@513] - EventThread shut down for session: 0x10020f038cf0000
2016-10-04 01:04:09,217 [myid:] - INFO [main:ClientBase@513] - STOPPING server
2016-10-04 01:04:09,218 [myid:] - INFO [main:NettyServerCnxnFactory@464] - shutdown called 0.0.0.0/0.0.0.0:16609
2016-10-04 01:04:09,224 [myid:] - INFO [main:ZooKeeperServer@529] - shutting down
2016-10-04 01:04:09,224 [myid:] - ERROR [main:ZooKeeperServer@501] - ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
2016-10-04 01:04:09,224 [myid:] - INFO [main:SessionTrackerImpl@232] - Shutting down
2016-10-04 01:04:09,225 [myid:] - INFO [main:PrepRequestProcessor@975] - Shutting down
2016-10-04 01:04:09,225 [myid:] - INFO [main:SyncRequestProcessor@191] - Shutting down
2016-10-04 01:04:09,225 [myid:] - INFO [ProcessThread(sid:0 cport:16609)::PrepRequestProcessor@154] - PrepRequestProcessor exited loop!
2016-10-04 01:04:09,225 [myid:] - INFO [SyncThread:0:SyncRequestProcessor@169] - SyncRequestProcessor exited!
2016-10-04 01:04:09,226 [myid:] - INFO [main:FinalRequestProcessor@479] - shutdown of request processor complete
2016-10-04 01:04:09,226 [myid:] - INFO [main:MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=StandaloneServer_port16609,name1=InMemoryDataTree]
2016-10-04 01:04:09,226 [myid:] - INFO [main:MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=StandaloneServer_port16609]
2016-10-04 01:04:09,227 [myid:] - INFO [main:FourLetterWordMain@85] - connecting to 127.0.0.1 16609
2016-10-04 01:04:09,228 [myid:] - INFO [main:JMXEnv@146] - ensureOnly:[]
2016-10-04 01:04:09,232 [myid:] - INFO [main:ClientBase@568] - fdcount after test is: 59 at start it was 60
2016-10-04 01:04:09,233 [myid:] - INFO [main:AsyncOpsTest@63] - Test clients shutting down
2016-10-04 01:04:09,234 [myid:] - INFO [main:ZKTestCase$1@65] - SUCCEEDED testAsyncMulti
2016-10-04 01:04:09,234 [myid:] - INFO [main:ZKTestCase$1@60] - FINISHED testAsyncMulti
2016-10-04 01:04:09,234 [myid:] - INFO [main:PortAssignment@85] - Assigned port 16610 from range 16607 - 19299.
2016-10-04 01:04:09,235 [myid:] - INFO [main:ZKTestCase$1@55] - STARTING testAsyncMultiSequential_NoSideEffect
2016-10-04 01:04:09,235 [myid:] - INFO [main:ClientBase@448] - Initial fdcount is: 59
2016-10-04 01:04:09,243 [myid:] - INFO [main:ClientBase@466] - STARTING server
2016-10-04 01:04:09,243 [myid:] - INFO [main:ClientBase@386] - CREATING server instance 127.0.0.1:16610
2016-10-04 01:04:09,250 [myid:] - INFO [main:ClientBase@361] - STARTING server instance 127.0.0.1:16610
2016-10-04 01:04:09,255 [myid:] - INFO [main:ZooKeeperServer@889] - minSessionTimeout set to 6000
2016-10-04 01:04:09,255 [myid:] - INFO [main:ZooKeeperServer@898] - maxSessionTimeout set to 60000
2016-10-04 01:04:09,255 [myid:] - INFO [main:ZooKeeperServer@159] - Created server with tickTime 3000 minSessionTimeout 6000 maxSessionTimeout 60000 datadir /home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/build/test/tmp/test7939377037580146364.junit.dir/version-2 snapdir /home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/build/test/tmp/test7939377037580146364.junit.dir/version-2
2016-10-04 01:04:09,256 [myid:] - INFO [main:NettyServerCnxnFactory@487] - binding to port 0.0.0.0/0.0.0.0:16610
2016-10-04 01:04:09,257 [myid:] - INFO [main:FileTxnSnapLog@306] - Snapshotting: 0x0 to /home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/build/test/tmp/test7939377037580146364.junit.dir/version-2/snapshot.0
2016-10-04 01:04:09,258 [myid:] - ERROR [main:ZooKeeperServer@501] - ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
2016-10-04 01:04:09,258 [myid:] - INFO [main:FourLetterWordMain@85] - connecting to 127.0.0.1 16610
2016-10-04 01:04:09,260 [myid:] - INFO [New I/O worker #133:NettyServerCnxn@275] - Processing stat command from /127.0.0.1:34416
2016-10-04 01:04:09,260 [myid:] - INFO [New I/O worker #133:StatCommand@49] - Stat command output
2016-10-04 01:04:09,261 [myid:] - INFO [main:JMXEnv@228] - ensureParent:[InMemoryDataTree, StandaloneServer_port]
2016-10-04 01:04:09,262 [myid:] - INFO [main:JMXEnv@245] - expect:InMemoryDataTree
2016-10-04 01:04:09,263 [myid:] - INFO [main:JMXEnv@249] - found:InMemoryDataTree org.apache.ZooKeeperService:name0=StandaloneServer_port16610,name1=InMemoryDataTree
2016-10-04 01:04:09,263 [myid:] - INFO [main:JMXEnv@245] - expect:StandaloneServer_port
2016-10-04 01:04:09,263 [myid:] - INFO [main:JMXEnv@249] - found:StandaloneServer_port org.apache.ZooKeeperService:name0=StandaloneServer_port16610
2016-10-04 01:04:09,264 [myid:] - INFO [main:ClientBase@462] - Client test setup finished
2016-10-04 01:04:09,264 [myid:] - INFO [main:AsyncOpsTest@50] - Creating client testAsyncMultiSequential_NoSideEffect
2016-10-04 01:04:09,264 [myid:] - INFO [main:ZooKeeper@853] - Initiating client connection, connectString=127.0.0.1:16610 sessionTimeout=30000 watcher=org.apache.zookeeper.test.ClientBase$CountdownWatcher@9a3da81
2016-10-04 01:04:09,266 [myid:127.0.0.1:16610] - INFO [main-SendThread(127.0.0.1:16610):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:16610. Will not attempt to authenticate using SASL (unknown error)
2016-10-04 01:04:09,266 [myid:127.0.0.1:16610] - INFO [main-SendThread(127.0.0.1:16610):ClientCnxn$SendThread@948] - Socket connection established, initiating session, client: /127.0.0.1:34417, server: 127.0.0.1/127.0.0.1:16610
2016-10-04 01:04:09,267 [myid:] - INFO [New I/O worker #134:ZooKeeperServer@995] - Client attempting to establish new session at /127.0.0.1:34417
2016-10-04 01:04:09,267 [myid:] - INFO [SyncThread:0:FileTxnLog@204] - Creating new log file: log.1
2016-10-04 01:04:09,305 [myid:] - INFO [SyncThread:0:ZooKeeperServer@709] - Established session 0x10020f039c90000 with negotiated timeout 30000 for client /127.0.0.1:34417
2016-10-04 01:04:09,305 [myid:127.0.0.1:16610] - INFO [main-SendThread(127.0.0.1:16610):ClientCnxn$SendThread@1381] - Session establishment complete on server 127.0.0.1/127.0.0.1:16610, sessionid = 0x10020f039c90000, negotiated timeout = 30000
2016-10-04 01:04:09,308 [myid:] - INFO [main:JMXEnv@117] - expect:0x10020f039c90000
2016-10-04 01:04:09,308 [myid:] - INFO [main:JMXEnv@120] - found:0x10020f039c90000 org.apache.ZooKeeperService:name0=StandaloneServer_port16610,name1=Connections,name2=127.0.0.1,name3=0x10020f039c90000
2016-10-04 01:04:09,309 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@77] - RUNNING TEST METHOD testAsyncMultiSequential_NoSideEffect
2016-10-04 01:04:09,309 [myid:] - INFO [New I/O worker #134:ZooKeeperServer@1032] - got auth packet /127.0.0.1:34417
2016-10-04 01:04:09,310 [myid:] - INFO [New I/O worker #134:ZooKeeperServer@1050] - auth success /127.0.0.1:34417
2016-10-04 01:04:09,384 [myid:] - INFO [ProcessThread(sid:0 cport:16610)::PrepRequestProcessor@790] - Got user-level KeeperException when processing sessionid:0x10020f039c90000 type:multi cxid:0x4 zxid:0x4 txntype:2 reqpath:n/a aborting remaining multi ops. Error Path:/nonexist Error:KeeperErrorCode = NoNode for /nonexist
2016-10-04 01:04:09,430 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@82] - Memory used 62828
2016-10-04 01:04:09,430 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@87] - Number of threads 56
2016-10-04 01:04:09,430 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@102] - FINISHED TEST METHOD testAsyncMultiSequential_NoSideEffect
2016-10-04 01:04:09,431 [myid:] - INFO [ProcessThread(sid:0 cport:16610)::PrepRequestProcessor@657] - Processed session termination for sessionid: 0x10020f039c90000
2016-10-04 01:04:09,454 [myid:] - WARN [New I/O worker #134:NettyServerCnxnFactory$CnxnChannelHandler@142] - Exception caught [id: 0x0743662e, /127.0.0.1:34417 => /127.0.0.1:16610] EXCEPTION: java.nio.channels.ClosedChannelException
java.nio.channels.ClosedChannelException
at sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:270)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:479)
at org.jboss.netty.channel.socket.nio.SocketSendBufferPool$UnpooledSendBuffer.transferTo(SocketSendBufferPool.java:203)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:201)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromTaskLoop(AbstractNioWorker.java:151)
at org.jboss.netty.channel.socket.nio.AbstractNioChannel$WriteTask.run(AbstractNioChannel.java:315)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:391)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:315)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2016-10-04 01:04:09,454 [myid:] - INFO [SyncThread:0:MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=StandaloneServer_port16610,name1=Connections,name2=127.0.0.1,name3=0x10020f039c90000]
2016-10-04 01:04:09,555 [myid:] - INFO [main:ZooKeeper@1311] - Session: 0x10020f039c90000 closed
2016-10-04 01:04:09,555 [myid:] - INFO [main:ClientBase@543] - tearDown starting
2016-10-04 01:04:09,555 [myid:] - INFO [main:ClientBase@513] - STOPPING server
2016-10-04 01:04:09,556 [myid:] - INFO [main:NettyServerCnxnFactory@464] - shutdown called 0.0.0.0/0.0.0.0:16610
2016-10-04 01:04:09,555 [myid:] - INFO [main-EventThread:ClientCnxn$EventThread@513] - EventThread shut down for session: 0x10020f039c90000
2016-10-04 01:04:09,562 [myid:] - INFO [main:ZooKeeperServer@529] - shutting down
2016-10-04 01:04:09,562 [myid:] - ERROR [main:ZooKeeperServer@501] - ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
2016-10-04 01:04:09,562 [myid:] - INFO [main:SessionTrackerImpl@232] - Shutting down
2016-10-04 01:04:09,563 [myid:] - INFO [main:PrepRequestProcessor@975] - Shutting down
2016-10-04 01:04:09,563 [myid:] - INFO [main:SyncRequestProcessor@191] - Shutting down
2016-10-04 01:04:09,563 [myid:] - INFO [ProcessThread(sid:0 cport:16610)::PrepRequestProcessor@154] - PrepRequestProcessor exited loop!
2016-10-04 01:0
...[truncated 6640749 chars]...
ting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:744)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:357)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1214)
2016-10-04 01:14:30,775 [myid:127.0.0.1:16699] - INFO [main-SendThread(127.0.0.1:16699):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:16699. Will not attempt to authenticate using SASL (unknown error)
2016-10-04 01:14:30,775 [myid:127.0.0.1:16699] - WARN [main-SendThread(127.0.0.1:16699):ClientCnxn$SendThread@1235] - Session 0x70020f2f1ed0001 for server 127.0.0.1/127.0.0.1:16699, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:744)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:357)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1214)
2016-10-04 01:14:30,826 [myid:127.0.0.1:16690] - INFO [main-SendThread(127.0.0.1:16690):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:16690. Will not attempt to authenticate using SASL (unknown error)
2016-10-04 01:14:30,826 [myid:127.0.0.1:16690] - WARN [main-SendThread(127.0.0.1:16690):ClientCnxn$SendThread@1235] - Session 0x40020f2e0da0001 for server 127.0.0.1/127.0.0.1:16690, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:744)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:357)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1214)
2016-10-04 01:14:30,912 [myid:] - INFO [SessionTracker:SessionTrackerImpl@158] - SessionTrackerImpl exited loop!
2016-10-04 01:14:30,913 [myid:] - INFO [SessionTracker:SessionTrackerImpl@158] - SessionTrackerImpl exited loop!
2016-10-04 01:14:30,913 [myid:] - INFO [SessionTracker:SessionTrackerImpl@158] - SessionTrackerImpl exited loop!
2016-10-04 01:14:30,981 [myid:127.0.0.1:16696] - INFO [main-SendThread(127.0.0.1:16696):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:16696. Will not attempt to authenticate using SASL (unknown error)
2016-10-04 01:14:30,982 [myid:127.0.0.1:16696] - WARN [main-SendThread(127.0.0.1:16696):ClientCnxn$SendThread@1235] - Session 0x60020f2ea050001 for server 127.0.0.1/127.0.0.1:16696, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:744)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:357)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1214)
2016-10-04 01:14:31,050 [myid:127.0.0.1:16681] - INFO [main-SendThread(127.0.0.1:16681):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:16681. Will not attempt to authenticate using SASL (unknown error)
2016-10-04 01:14:31,051 [myid:127.0.0.1:16681] - WARN [main-SendThread(127.0.0.1:16681):ClientCnxn$SendThread@1235] - Session 0x10020f2dfad0001 for server 127.0.0.1/127.0.0.1:16681, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:744)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:357)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1214)
2016-10-04 01:14:31,051 [myid:127.0.0.1:16687] - INFO [main-SendThread(127.0.0.1:16687):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:16687. Will not attempt to authenticate using SASL (unknown error)
2016-10-04 01:14:31,051 [myid:127.0.0.1:16687] - WARN [main-SendThread(127.0.0.1:16687):ClientCnxn$SendThread@1235] - Session 0x30020f2dfad0001 for server 127.0.0.1/127.0.0.1:16687, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:744)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:357)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1214)
2016-10-04 01:14:31,073 [myid:127.0.0.1:16693] - INFO [main-SendThread(127.0.0.1:16693):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:16693. Will not attempt to authenticate using SASL (unknown error)
2016-10-04 01:14:31,074 [myid:127.0.0.1:16693] - WARN [main-SendThread(127.0.0.1:16693):ClientCnxn$SendThread@1235] - Session 0x50020f2e0e30001 for server 127.0.0.1/127.0.0.1:16693, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:744)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:357)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1214)
2016-10-04 01:14:31,152 [myid:127.0.0.1:16729] - INFO [main-SendThread(127.0.0.1:16729):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:16729. Will not attempt to authenticate using SASL (unknown error)
2016-10-04 01:14:31,152 [myid:127.0.0.1:16729] - WARN [main-SendThread(127.0.0.1:16729):ClientCnxn$SendThread@1235] - Session 0x10020f685ea0000 for server 127.0.0.1/127.0.0.1:16729, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:744)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:357)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1214)
2016-10-04 01:14:31,190 [myid:127.0.0.1:16681] - INFO [main-SendThread(127.0.0.1:16681):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:16681. Will not attempt to authenticate using SASL (unknown error)
2016-10-04 01:14:31,190 [myid:127.0.0.1:16681] - WARN [main-SendThread(127.0.0.1:16681):ClientCnxn$SendThread@1235] - Session 0x10020f2dfad0000 for server 127.0.0.1/127.0.0.1:16681, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:744)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:357)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1214)
2016-10-04 01:14:31,315 [myid:] - INFO [main:ClientBase@466] - STARTING server
2016-10-04 01:14:31,315 [myid:] - INFO [main:ClientBase@386] - CREATING server instance 127.0.0.1:16852
2016-10-04 01:14:31,324 [myid:] - INFO [main:ClientBase@361] - STARTING server instance 127.0.0.1:16852
2016-10-04 01:14:31,324 [myid:] - INFO [main:ZooKeeperServer@889] - minSessionTimeout set to 6000
2016-10-04 01:14:31,324 [myid:] - INFO [main:ZooKeeperServer@898] - maxSessionTimeout set to 60000
2016-10-04 01:14:31,324 [myid:] - INFO [main:ZooKeeperServer@159] - Created server with tickTime 3000 minSessionTimeout 6000 maxSessionTimeout 60000 datadir /home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/build/test/tmp/test8970971002755443544.junit.dir/version-2 snapdir /home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/build/test/tmp/test8970971002755443544.junit.dir/version-2
2016-10-04 01:14:31,324 [myid:] - INFO [main:NettyServerCnxnFactory@487] - binding to port 0.0.0.0/0.0.0.0:16852
2016-10-04 01:14:31,325 [myid:] - INFO [main:FileSnap@83] - Reading snapshot /home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/build/test/tmp/test8970971002755443544.junit.dir/version-2/snapshot.3
2016-10-04 01:14:31,326 [myid:] - INFO [main:FileTxnSnapLog@306] - Snapshotting: 0x5 to /home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/build/test/tmp/test8970971002755443544.junit.dir/version-2/snapshot.5
2016-10-04 01:14:31,327 [myid:] - ERROR [main:ZooKeeperServer@501] - ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
2016-10-04 01:14:31,328 [myid:] - INFO [main:FourLetterWordMain@85] - connecting to 127.0.0.1 16852
2016-10-04 01:14:31,329 [myid:] - INFO [New I/O worker #6601:NettyServerCnxn@275] - Processing stat command from /127.0.0.1:38348
2016-10-04 01:14:31,329 [myid:] - INFO [New I/O worker #6601:StatCommand@49] - Stat command output
2016-10-04 01:14:31,329 [myid:] - INFO [main:JMXEnv@228] - ensureParent:[InMemoryDataTree, StandaloneServer_port]
2016-10-04 01:14:31,331 [myid:] - INFO [main:JMXEnv@245] - expect:InMemoryDataTree
2016-10-04 01:14:31,331 [myid:] - INFO [main:JMXEnv@249] - found:InMemoryDataTree org.apache.ZooKeeperService:name0=StandaloneServer_port16852,name1=InMemoryDataTree
2016-10-04 01:14:31,331 [myid:] - INFO [main:JMXEnv@245] - expect:StandaloneServer_port
2016-10-04 01:14:31,331 [myid:] - INFO [main:JMXEnv@249] - found:StandaloneServer_port org.apache.ZooKeeperService:name0=StandaloneServer_port16852
2016-10-04 01:14:31,433 [myid:127.0.0.1:16699] - INFO [main-SendThread(127.0.0.1:16699):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:16699. Will not attempt to authenticate using SASL (unknown error)
2016-10-04 01:14:31,434 [myid:127.0.0.1:16699] - WARN [main-SendThread(127.0.0.1:16699):ClientCnxn$SendThread@1235] - Session 0x70020f2f1ed0000 for server 127.0.0.1/127.0.0.1:16699, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:744)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:357)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1214)
2016-10-04 01:14:31,554 [myid:127.0.0.1:16684] - INFO [main-SendThread(127.0.0.1:16684):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:16684. Will not attempt to authenticate using SASL (unknown error)
2016-10-04 01:14:31,554 [myid:127.0.0.1:16684] - WARN [main-SendThread(127.0.0.1:16684):ClientCnxn$SendThread@1235] - Session 0x20020f2dfad0000 for server 127.0.0.1/127.0.0.1:16684, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:744)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:357)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1214)
2016-10-04 01:14:31,673 [myid:127.0.0.1:16732] - INFO [main-SendThread(127.0.0.1:16732):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:16732. Will not attempt to authenticate using SASL (unknown error)
2016-10-04 01:14:31,674 [myid:127.0.0.1:16732] - WARN [main-SendThread(127.0.0.1:16732):ClientCnxn$SendThread@1235] - Session 0x20020f689b10000 for server 127.0.0.1/127.0.0.1:16732, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:744)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:357)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1214)
2016-10-04 01:14:31,865 [myid:127.0.0.1:16735] - INFO [main-SendThread(127.0.0.1:16735):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:16735. Will not attempt to authenticate using SASL (unknown error)
2016-10-04 01:14:31,865 [myid:127.0.0.1:16735] - WARN [main-SendThread(127.0.0.1:16735):ClientCnxn$SendThread@1235] - Session 0x30020f6861c0000 for server 127.0.0.1/127.0.0.1:16735, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:744)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:357)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1214)
2016-10-04 01:14:31,926 [myid:127.0.0.1:16684] - INFO [main-SendThread(127.0.0.1:16684):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:16684. Will not attempt to authenticate using SASL (unknown error)
2016-10-04 01:14:31,926 [myid:127.0.0.1:16684] - WARN [main-SendThread(127.0.0.1:16684):ClientCnxn$SendThread@1235] - Session 0x20020f2dfad0001 for server 127.0.0.1/127.0.0.1:16684, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:744)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:357)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1214)
2016-10-04 01:14:31,964 [myid:127.0.0.1:16693] - INFO [main-SendThread(127.0.0.1:16693):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:16693. Will not attempt to authenticate using SASL (unknown error)
2016-10-04 01:14:31,964 [myid:127.0.0.1:16693] - WARN [main-SendThread(127.0.0.1:16693):ClientCnxn$SendThread@1235] - Session 0x50020f2e0e30000 for server 127.0.0.1/127.0.0.1:16693, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:744)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:357)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1214)
2016-10-04 01:14:32,120 [myid:127.0.0.1:16608] - INFO [main-SendThread(127.0.0.1:16608):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:16608. Will not attempt to authenticate using SASL (unknown error)
2016-10-04 01:14:32,120 [myid:127.0.0.1:16608] - WARN [main-SendThread(127.0.0.1:16608):ClientCnxn$SendThread@1235] - Session 0x10020f0235b0000 for server 127.0.0.1/127.0.0.1:16608, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:744)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:357)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1214)
2016-10-04 01:14:32,158 [myid:127.0.0.1:16690] - INFO [main-SendThread(127.0.0.1:16690):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:16690. Will not attempt to authenticate using SASL (unknown error)
2016-10-04 01:14:32,158 [myid:127.0.0.1:16690] - WARN [main-SendThread(127.0.0.1:16690):ClientCnxn$SendThread@1235] - Session 0x40020f2e0da0000 for server 127.0.0.1/127.0.0.1:16690, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:744)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:357)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1214)
2016-10-04 01:14:32,201 [myid:127.0.0.1:16852] - INFO [main-SendThread(127.0.0.1:16852):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:16852. Will not attempt to authenticate using SASL (unknown error)
2016-10-04 01:14:32,201 [myid:127.0.0.1:16852] - INFO [main-SendThread(127.0.0.1:16852):ClientCnxn$SendThread@948] - Socket connection established, initiating session, client: /127.0.0.1:38382, server: 127.0.0.1/127.0.0.1:16852
2016-10-04 01:14:32,202 [myid:] - INFO [New I/O worker #6602:ZooKeeperServer@1000] - Client attempting to renew session 0x10020f9ae620000 at /127.0.0.1:38382
2016-10-04 01:14:32,203 [myid:] - INFO [New I/O worker #6602:ZooKeeperServer@709] - Established session 0x10020f9ae620000 with negotiated timeout 6000 for client /127.0.0.1:38382
2016-10-04 01:14:32,203 [myid:127.0.0.1:16852] - INFO [main-SendThread(127.0.0.1:16852):ClientCnxn$SendThread@1381] - Session establishment complete on server 127.0.0.1/127.0.0.1:16852, sessionid = 0x10020f9ae620000, negotiated timeout = 6000
2016-10-04 01:14:32,210 [myid:127.0.0.1:16687] - INFO [main-SendThread(127.0.0.1:16687):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:16687. Will not attempt to authenticate using SASL (unknown error)
2016-10-04 01:14:32,210 [myid:127.0.0.1:16687] - WARN [main-SendThread(127.0.0.1:16687):ClientCnxn$SendThread@1235] - Session 0x30020f2dfad0001 for server 127.0.0.1/127.0.0.1:16687, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:744)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:357)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1214)
2016-10-04 01:14:32,210 [myid:] - INFO [SyncThread:0:FileTxnLog@204] - Creating new log file: log.6
2016-10-04 01:14:32,248 [myid:] - INFO [main:ClientBase@513] - STOPPING server
2016-10-04 01:14:32,248 [myid:] - INFO [main:NettyServerCnxnFactory@464] - shutdown called 0.0.0.0/0.0.0.0:16852
2016-10-04 01:14:32,248 [myid:127.0.0.1:16852] - INFO [main-SendThread(127.0.0.1:16852):ClientCnxn$SendThread@1231] - Unable to read additional data from server sessionid 0x10020f9ae620000, likely server has closed socket, closing socket connection and attempting reconnect
2016-10-04 01:14:32,248 [myid:] - INFO [main:MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=StandaloneServer_port16852,name1=Connections,name2=127.0.0.1,name3=0x10020f9ae620000]
2016-10-04 01:14:32,254 [myid:] - INFO [main:ZooKeeperServer@529] - shutting down
2016-10-04 01:14:32,254 [myid:] - ERROR [main:ZooKeeperServer@501] - ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
2016-10-04 01:14:32,254 [myid:] - INFO [main:SessionTrackerImpl@232] - Shutting down
2016-10-04 01:14:32,254 [myid:] - INFO [main:PrepRequestProcessor@975] - Shutting down
2016-10-04 01:14:32,254 [myid:] - INFO [main:SyncRequestProcessor@191] - Shutting down
2016-10-04 01:14:32,254 [myid:] - INFO [ProcessThread(sid:0 cport:16852)::PrepRequestProcessor@154] - PrepRequestProcessor exited loop!
2016-10-04 01:14:32,255 [myid:] - INFO [SyncThread:0:SyncRequestProcessor@169] - SyncRequestProcessor exited!
2016-10-04 01:14:32,255 [myid:] - INFO [main:FinalRequestProcessor@479] - shutdown of request processor complete
2016-10-04 01:14:32,255 [myid:] - INFO [main:MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=StandaloneServer_port16852,name1=InMemoryDataTree]
2016-10-04 01:14:32,255 [myid:] - INFO [main:MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=StandaloneServer_port16852]
2016-10-04 01:14:32,256 [myid:] - INFO [main:FourLetterWordMain@85] - connecting to 127.0.0.1 16852
2016-10-04 01:14:32,256 [myid:] - INFO [main:JMXEnv@146] - ensureOnly:[]
2016-10-04 01:14:32,349 [myid:] - INFO [main:ClientBase@466] - STARTING server
2016-10-04 01:14:32,349 [myid:] - INFO [main:ClientBase@386] - CREATING server instance 127.0.0.1:16852
2016-10-04 01:14:32,356 [myid:] - INFO [main:ClientBase@361] - STARTING server instance 127.0.0.1:16852
2016-10-04 01:14:32,356 [myid:] - INFO [main:ZooKeeperServer@889] - minSessionTimeout set to 6000
2016-10-04 01:14:32,356 [myid:] - INFO [main:ZooKeeperServer@898] - maxSessionTimeout set to 60000
2016-10-04 01:14:32,356 [myid:] - INFO [main:ZooKeeperServer@159] - Created server with tickTime 3000 minSessionTimeout 6000 maxSessionTimeout 60000 datadir /home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/build/test/tmp/test8970971002755443544.junit.dir/version-2 snapdir /home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/build/test/tmp/test8970971002755443544.junit.dir/version-2
2016-10-04 01:14:32,356 [myid:] - INFO [main:NettyServerCnxnFactory@487] - binding to port 0.0.0.0/0.0.0.0:16852
2016-10-04 01:14:32,357 [myid:] - INFO [main:FileSnap@83] - Reading snapshot /home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/build/test/tmp/test8970971002755443544.junit.dir/version-2/snapshot.5
2016-10-04 01:14:32,358 [myid:] - INFO [main:FileTxnSnapLog@306] - Snapshotting: 0x6 to /home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/build/test/tmp/test8970971002755443544.junit.dir/version-2/snapshot.6
2016-10-04 01:14:32,359 [myid:] - ERROR [main:ZooKeeperServer@501] - ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
2016-10-04 01:14:32,359 [myid:] - INFO [main:FourLetterWordMain@85] - connecting to 127.0.0.1 16852
2016-10-04 01:14:32,360 [myid:] - INFO [New I/O worker #6634:NettyServerCnxn@275] - Processing stat command from /127.0.0.1:38388
2016-10-04 01:14:32,360 [myid:] - INFO [New I/O worker #6634:StatCommand@49] - Stat command output
2016-10-04 01:14:32,361 [myid:] - INFO [main:JMXEnv@228] - ensureParent:[InMemoryDataTree, StandaloneServer_port]
2016-10-04 01:14:32,362 [myid:] - INFO [main:JMXEnv@245] - expect:InMemoryDataTree
2016-10-04 01:14:32,362 [myid:] - INFO [main:JMXEnv@249] - found:InMemoryDataTree org.apache.ZooKeeperService:name0=StandaloneServer_port16852,name1=InMemoryDataTree
2016-10-04 01:14:32,362 [myid:] - INFO [main:JMXEnv@245] - expect:StandaloneServer_port
2016-10-04 01:14:32,362 [myid:] - INFO [main:JMXEnv@249] - found:StandaloneServer_port org.apache.ZooKeeperService:name0=StandaloneServer_port16852
2016-10-04 01:14:32,422 [myid:127.0.0.1:16687] - INFO [main-SendThread(127.0.0.1:16687):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:16687. Will not attempt to authenticate using SASL (unknown error)
2016-10-04 01:14:32,422 [myid:127.0.0.1:16687] - WARN [main-SendThread(127.0.0.1:16687):ClientCnxn$SendThread@1235] - Session 0x30020f2dfad0000 for server 127.0.0.1/127.0.0.1:16687, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:744)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:357)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1214)
2016-10-04 01:14:32,527 [myid:127.0.0.1:16696] - INFO [main-SendThread(127.0.0.1:16696):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:16696. Will not attempt to authenticate using SASL (unknown error)
2016-10-04 01:14:32,528 [myid:127.0.0.1:16696] - WARN [main-SendThread(127.0.0.1:16696):ClientCnxn$SendThread@1235] - Session 0x60020f2ea050001 for server 127.0.0.1/127.0.0.1:16696, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:744)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:357)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1214)
2016-10-04 01:14:32,580 [myid:127.0.0.1:16699] - INFO [main-SendThread(127.0.0.1:16699):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:16699. Will not attempt to authenticate using SASL (unknown error)
2016-10-04 01:14:32,581 [myid:127.0.0.1:16699] - WARN [main-SendThread(127.0.0.1:16699):ClientCnxn$SendThread@1235] - Session 0x70020f2f1ed0000 for server 127.0.0.1/127.0.0.1:16699, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:744)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:357)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1214)
2016-10-04 01:14:32,718 [myid:127.0.0.1:16699] - INFO [main-SendThread(127.0.0.1:16699):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:16699. Will not attempt to authenticate using SASL (unknown error)
2016-10-04 01:14:32,718 [myid:127.0.0.1:16699] - WARN [main-SendThread(127.0.0.1:16699):ClientCnxn$SendThread@1235] - Session 0x70020f2f1ed0001 for server 127.0.0.1/127.0.0.1:16699, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:744)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:357)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1214)
2016-10-04 01:14:32,737 [myid:127.0.0.1:16696] - INFO [main-SendThread(127.0.0.1:16696):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:16696. Will not attempt to authenticate using SASL (unknown error)
2016-10-04 01:14:32,737 [myid:127.0.0.1:16696] - WARN [main-SendThread(127.0.0.1:16696):ClientCnxn$SendThread@1235] - Session 0x60020f2ea050000 for server 127.0.0.1/127.0.0.1:16696, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:744)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:357)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1214)
2016-10-04 01:14:32,791 [myid:127.0.0.1:16690] - INFO [main-SendThread(127.0.0.1:16690):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:16690. Will not attempt to authenticate using SASL (unknown error)
2016-10-04 01:14:32,791 [myid:127.0.0.1:16690] - WARN [main-SendThread(127.0.0.1:16690):ClientCnxn$SendThread@1235] - Session 0x40020f2e0da0001 for server 127.0.0.1/127.0.0.1:16690, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:744)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:357)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1214)
2016-10-04 01:14:32,853 [myid:127.0.0.1:16729] - INFO [main-SendThread(127.0.0.1:16729):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:16729. Will not attempt to authenticate using SASL (unknown error)
2016-10-04 01:14:32,853 [myid:127.0.0.1:16729] - WARN [main-SendThread(127.0.0.1:16729):ClientCnxn$SendThread@1235] - Session 0x10020f685ea0000 for server 127.0.0.1/127.0.0.1:16729, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:744)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:357)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1214)
2016-10-04 01:14:32,913 [myid:127.0.0.1:16681] - INFO [main-SendThread(127.0.0.1:16681):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:16681. Will not attempt to authenticate using SASL (unknown error)
2016-10-04 01:14:32,913 [myid:127.0.0.1:16681] - WARN [main-SendThread(127.0.0.1:16681):ClientCnxn$SendThread@1235] - Session 0x10020f2dfad0000 for server 127.0.0.1/127.0.0.1:16681, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:744)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:357)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1214)
2016-10-04 01:14:32,975 [myid:127.0.0.1:16684] - INFO [main-SendThread(127.0.0.1:16684):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:16684. Will not attempt to authenticate using SASL (unknown error)
2016-10-04 01:14:32,975 [myid:127.0.0.1:16684] - WARN [main-SendThread(127.0.0.1:16684):ClientCnxn$SendThread@1235] - Session 0x20020f2dfad0000 for server 127.0.0.1/127.0.0.1:16684, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:744)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:357)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1214)
2016-10-04 01:14:33,037 [myid:127.0.0.1:16684] - INFO [main-SendThread(127.0.0.1:16684):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:16684. Will not attempt to authenticate using SASL (unknown error)
2016-10-04 01:14:33,037 [myid:127.0.0.1:16684] - WARN [main-SendThread(127.0.0.1:16684):ClientCnxn$SendThread@1235] - Session 0x20020f2dfad0001 for server 127.0.0.1/127.0.0.1:16684, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:744)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:357)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1214)
2016-10-04 01:14:33,077 [myid:127.0.0.1:16681] - INFO [main-SendThread(127.0.0.1:16681):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:16681. Will not attempt to authenticate using SASL (unknown error)
2016-10-04 01:14:33,078 [myid:127.0.0.1:16681] - WARN [main-SendThread(127.0.0.1:16681):ClientCnxn$SendThread@1235] - Session 0x10020f2dfad0001 for server 127.0.0.1/127.0.0.1:16681, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:744)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:357)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1214)
2016-10-04 01:14:33,113 [myid:127.0.0.1:16693] - INFO [main-SendThread(127.0.0.1:16693):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:16693. Will not attempt to authenticate using SASL (unknown error)
2016-10-04 01:14:33,114 [myid:127.0.0.1:16693] - WARN [main-SendThread(127.0.0.1:16693):ClientCnxn$SendThread@1235] - Session 0x50020f2e0e30001 for server 127.0.0.1/127.0.0.1:16693, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:744)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:357)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1214)
2016-10-04 01:14:33,128 [myid:127.0.0.1:16693] - INFO [main-SendThread(127.0.0.1:16693):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:16693. Will not attempt to authenticate using SASL (unknown error)
2016-10-04 01:14:33,130 [myid:127.0.0.1:16693] - WARN [main-SendThread(127.0.0.1:16693):ClientCnxn$SendThread@1235] - Session 0x50020f2e0e30000 for server 127.0.0.1/127.0.0.1:16693, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:744)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:357)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1214)
2016-10-04 01:14:33,133 [myid:127.0.0.1:16732] - INFO [main-SendThread(127.0.0.1:16732):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:16732. Will not attempt to authenticate using SASL (unknown error)
2016-10-04 01:14:33,134 [myid:127.0.0.1:16732] - WARN [main-SendThread(127.0.0.1:16732):ClientCnxn$SendThread@1235] - Session 0x20020f689b10000 for server 127.0.0.1/127.0.0.1:16732, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:744)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:357)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1214)
2016-10-04 01:14:33,361 [myid:127.0.0.1:16852] - INFO [main-SendThread(127.0.0.1:16852):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:16852. Will not attempt to authenticate using SASL (unknown error)
2016-10-04 01:14:33,362 [myid:127.0.0.1:16852] - INFO [main-SendThread(127.0.0.1:16852):ClientCnxn$SendThread@948] - Socket connection established, initiating session, client: /127.0.0.1:38420, server: 127.0.0.1/127.0.0.1:16852
2016-10-04 01:14:33,362 [myid:] - INFO [New I/O worker #6635:ZooKeeperServer@1000] - Client attempting to renew session 0x10020f9ae620000 at /127.0.0.1:38420
2016-10-04 01:14:33,363 [myid:] - INFO [New I/O worker #6635:ZooKeeperServer@709] - Established session 0x10020f9ae620000 with negotiated timeout 6000 for client /127.0.0.1:38420
2016-10-04 01:14:33,363 [myid:127.0.0.1:16852] - INFO [main-SendThread(127.0.0.1:16852):ClientCnxn$SendThread@1381] - Session establishment complete on server 127.0.0.1/127.0.0.1:16852, sessionid = 0x10020f9ae620000, negotiated timeout = 6000
2016-10-04 01:14:33,368 [myid:] - INFO [SyncThread:0:FileTxnLog@204] - Creating new log file: log.7
2016-10-04 01:14:33,548 [myid:127.0.0.1:16608] - INFO [main-SendThread(127.0.0.1:16608):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:16608. Will not attempt to authenticate using SASL (unknown error)
2016-10-04 01:14:33,548 [myid:127.0.0.1:16608] - WARN [main-SendThread(127.0.0.1:16608):ClientCnxn$SendThread@1235] - Session 0x10020f0235b0000 for server 127.0.0.1/127.0.0.1:16608, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:744)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:357)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1214)
2016-10-04 01:14:33,724 [myid:127.0.0.1:16699] - INFO [main-SendThread(127.0.0.1:16699):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:16699. Will not attempt to authenticate using SASL (unknown error)
2016-10-04 01:14:33,724 [myid:127.0.0.1:16699] - WARN [main-SendThread(127.0.0.1:16699):ClientCnxn$SendThread@1235] - Session 0x70020f2f1ed0000 for server 127.0.0.1/127.0.0.1:16699, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:744)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:357)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1214)
2016-10-04 01:14:33,875 [myid:127.0.0.1:16690] - INFO [main-SendThread(127.0.0.1:16690):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:16690. Will not attempt to authenticate using SASL (unknown error)
2016-10-04 01:14:33,875 [myid:127.0.0.1:16690] - WARN [main-SendThread(127.0.0.1:16690):ClientCnxn$SendThread@1235] - Session 0x40020f2e0da0000 for server 127.0.0.1/127.0.0.1:16690, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:744)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:357)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1214)
2016-10-04 01:14:33,881 [myid:127.0.0.1:16735] - INFO [main-SendThread(127.0.0.1:16735):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:16735. Will not attempt to authenticate using SASL (unknown error)
2016-10-04 01:14:33,881 [myid:127.0.0.1:16735] - WARN [main-SendThread(127.0.0.1:16735):ClientCnxn$SendThread@1235] - Session 0x30020f6861c0000 for server 127.0.0.1/127.0.0.1:16735, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:744)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:357)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1214)
2016-10-04 01:14:33,899 [myid:127.0.0.1:16696] - INFO [main-SendThread(127.0.0.1:16696):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:16696. Will not attempt to authenticate using SASL (unknown error)
2016-10-04 01:14:33,899 [myid:127.0.0.1:16696] - WARN [main-SendThread(127.0.0.1:16696):ClientCnxn$SendThread@1235] - Session 0x60020f2ea050000 for server 127.0.0.1/127.0.0.1:16696, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:744)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:357)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1214)
2016-10-04 01:14:33,912 [myid:] - INFO [SessionTracker:SessionTrackerImpl@158] - SessionTrackerImpl exited loop!
2016-10-04 01:14:33,929 [myid:127.0.0.1:16696] - INFO [main-SendThread(127.0.0.1:16696):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:16696. Will not attempt to authenticate using SASL (unknown error)
2016-10-04 01:14:33,930 [myid:127.0.0.1:16696] - WARN [main-SendThread(127.0.0.1:16696):ClientCnxn$SendThread@1235] - Session 0x60020f2ea050001 for server 127.0.0.1/127.0.0.1:16696, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:744)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:357)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1214)
2016-10-04 01:14:34,202 [myid:127.0.0.1:16687] - INFO [main-SendThread(127.0.0.1:16687):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:16687. Will not attempt to authenticate using SASL (unknown error)
2016-10-04 01:14:34,202 [myid:127.0.0.1:16687] - WARN [main-SendThread(127.0.0.1:16687):ClientCnxn$SendThread@1235] - Session 0x30020f2dfad0000 for server 127.0.0.1/127.0.0.1:16687, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:744)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:357)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1214)
2016-10-04 01:14:34,233 [myid:127.0.0.1:16687] - INFO [main-SendThread(127.0.0.1:16687):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:16687. Will not attempt to authenticate using SASL (unknown error)
2016-10-04 01:14:34,233 [myid:127.0.0.1:16687] - WARN [main-SendThread(127.0.0.1:16687):ClientCnxn$SendThread@1235] - Session 0x30020f2dfad0001 for server 127.0.0.1/127.0.0.1:16687, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:744)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:357)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1214)
2016-10-04 01:14:34,240 [myid:127.0.0.1:16693] - INFO [main-SendThread(127.0.0.1:16693):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:16693. Will not attempt to authenticate using SASL (unknown error)
2016-10-04 01:14:34,241 [myid:127.0.0.1:16693] - WARN [main-SendThread(127.0.0.1:16693):ClientCnxn$SendThread@1235] - Session 0x50020f2e0e30000 for server 127.0.0.1/127.0.0.1:16693, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:744)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:357)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1214)
2016-10-04 01:14:34,349 [myid:127.0.0.1:16681] - INFO [main-SendThread(127.0.0.1:16681):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:16681. Will not attempt to authenticate using SASL (unknown error)
2016-10-04 01:14:34,350 [myid:127.0.0.1:16681] - WARN [main-SendThread(127.0.0.1:16681):ClientCnxn$SendThread@1235] - Session 0x10020f2dfad0001 for server 127.0.0.1/127.0.0.1:16681, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:744)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:357)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1214)
2016-10-04 01:14:34,424 [myid:] - INFO [ProcessThread(sid:0 cport:16852)::PrepRequestProcessor@657] - Processed session termination for sessionid: 0x10020f9ae620000
2016-10-04 01:14:34,432 [myid:] - WARN [New I/O worker #6635:NettyServerCnxnFactory$CnxnChannelHandler@142] - Exception caught [id: 0xfb034ea1, /127.0.0.1:38420 :> /127.0.0.1:16852] EXCEPTION: java.nio.channels.ClosedChannelException
java.nio.channels.ClosedChannelException
at sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:270)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:479)
at org.jboss.netty.channel.socket.nio.SocketSendBufferPool$UnpooledSendBuffer.transferTo(SocketSendBufferPool.java:203)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:201)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromTaskLoop(AbstractNioWorker.java:151)
at org.jboss.netty.channel.socket.nio.AbstractNioChannel$WriteTask.run(AbstractNioChannel.java:315)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:391)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:315)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2016-10-04 01:14:34,432 [myid:] - INFO [SyncThread:0:MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=StandaloneServer_port16852,name1=Connections,name2=127.0.0.1,name3=0x10020f9ae620000]
2016-10-04 01:14:34,473 [myid:127.0.0.1:16729] - INFO [main-SendThread(127.0.0.1:16729):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:16729. Will not attempt to authenticate using SASL (unknown error)
2016-10-04 01:14:34,474 [myid:127.0.0.1:16729] - WARN [main-SendThread(127.0.0.1:16729):ClientCnxn$SendThread@1235] - Session 0x10020f685ea0000 for server 127.0.0.1/127.0.0.1:16729, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:744)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:357)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1214)
2016-10-04 01:14:34,532 [myid:] - INFO [main:ZooKeeper@1311] - Session: 0x10020f9ae620000 closed
2016-10-04 01:14:34,532 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@82] - Memory used 187485
2016-10-04 01:14:34,532 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@87] - Number of threads 1671
2016-10-04 01:14:34,533 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@102] - FINISHED TEST METHOD testWatcherAutoResetWithLocal
2016-10-04 01:14:34,533 [myid:] - INFO [main:ClientBase@543] - tearDown starting
2016-10-04 01:14:34,533 [myid:] - INFO [main:ClientBase@513] - STOPPING server
2016-10-04 01:14:34,533 [myid:] - INFO [main:NettyServerCnxnFactory@464] - shutdown called 0.0.0.0/0.0.0.0:16852
2016-10-04 01:14:34,532 [myid:] - INFO [main-EventThread:ClientCnxn$EventThread@513] - EventThread shut down for session: 0x10020f9ae620000
2016-10-04 01:14:34,538 [myid:] - INFO [main:ZooKeeperServer@529] - shutting down
2016-10-04 01:14:34,539 [myid:] - ERROR [main:ZooKeeperServer@501] - ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
2016-10-04 01:14:34,539 [myid:] - INFO [main:SessionTrackerImpl@232] - Shutting down
2016-10-04 01:14:34,539 [myid:] - INFO [main:PrepRequestProcessor@975] - Shutting down
2016-10-04 01:14:34,539 [myid:] - INFO [main:SyncRequestProcessor@191] - Shutting down
2016-10-04 01:14:34,539 [myid:] - INFO [ProcessThread(sid:0 cport:16852)::PrepRequestProcessor@154] - PrepRequestProcessor exited loop!
2016-10-04 01:14:34,539 [myid:] - INFO [SyncThread:0:SyncRequestProcessor@169] - SyncRequestProcessor exited!
2016-10-04 01:14:34,539 [myid:] - INFO [main:FinalRequestProcessor@479] - shutdown of request processor complete
2016-10-04 01:14:34,540 [myid:] - INFO [main:MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=StandaloneServer_port16852,name1=InMemoryDataTree]
2016-10-04 01:14:34,540 [myid:] - INFO [main:MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=StandaloneServer_port16852]
2016-10-04 01:14:34,540 [myid:] - INFO [main:FourLetterWordMain@85] - connecting to 127.0.0.1 16852
2016-10-04 01:14:34,541 [myid:] - INFO [main:JMXEnv@146] - ensureOnly:[]
2016-10-04 01:14:34,547 [myid:] - INFO [main:ClientBase@568] - fdcount after test is: 4887 at start it was 4887
2016-10-04 01:14:34,547 [myid:] - INFO [main:ZKTestCase$1@65] - SUCCEEDED testWatcherAutoResetWithLocal
2016-10-04 01:14:34,547 [myid:] - INFO [main:ZKTestCase$1@60] - FINISHED testWatcherAutoResetWithLocal
{noformat}
flaky, flaky-test 9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
1 year, 21 weeks ago 0|i34mg7:
ZooKeeper ZOOKEEPER-2609

ZOOKEEPER-2169 Add TTL Node APIs to C client

Sub-task Closed Major Fixed Balazs Meszaros Jordan Zimmerman Jordan Zimmerman 08/Oct/16 12:59   16/Oct/19 14:59 09/Apr/19 05:39   3.6.0, 3.5.6 c client, java client, jute, server   0 2 0 1800   Need to update the C lib to have the TTL node option 100% 100% 1800 0 pull-request-available, ttl_nodes 9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
49 weeks, 2 days ago 0|i34mcn:
ZooKeeper ZOOKEEPER-2608

ZOOKEEPER-2169 Create CLI option for TTL ephemerals

Sub-task Resolved Major Fixed Jordan Zimmerman Camille Fournier Camille Fournier 07/Oct/16 15:40   21/Jan/19 08:41 23/Mar/17 13:47   3.6.0 c client, java client, jute, server   0 5   Need to update CreateCommand to have the TTL node option ttl_nodes 9223372036854775807 No Perforce job exists for this issue. 3 9223372036854775807
3 years ago 0|i34ldr:
ZooKeeper ZOOKEEPER-2607

doxygen-related, ./configure fails

Bug Open Major Unresolved Unassigned HIthu Anand HIthu Anand 07/Oct/16 03:17   07/Oct/16 03:24           0 2   Ubuntu14.04 linux 4.4.0-38-generic #57~14.04.1-Ubuntu SMP Tue Sep 6 17:20:43 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux hithu@linux:~/opt/gridlabd-src-2_0_2363$ autoreconf -isf
configure.ac:95: error: AC_SUBST: `DX_FLAG_[]DX_CURRENT_FEATURE' is not a valid shell variable name
m4/dx_doxygen.m4:77: DX_REQUIRE_PROG is expanded from...
m4/dx_doxygen.m4:117: DX_ARG_ABLE is expanded from...
m4/dx_doxygen.m4:178: DX_INIT_DOXYGEN is expanded from...
configure.ac:95: the top level
autom4te: /usr/bin/m4 failed with exit status: 1
aclocal: error: echo failed with exit status: 1
autoreconf: aclocal failed with exit status: 1


hithu@linux:~$ doxygen --version
1.8.6
hithu@linux:~$ autoconf --version
autoconf (GNU Autoconf) 2.69

hithu@linux:~$ automake --version
automake (GNU automake) 1.14.1
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 23 weeks, 6 days ago 0|i34k5j:
ZooKeeper ZOOKEEPER-2606

SaslServerCallbackHandler#handleAuthorizeCallback() should log the exception

Bug Closed Minor Fixed Ted Yu Ted Yu Ted Yu 04/Oct/16 09:00   31/Mar/17 05:01 17/Oct/16 04:48   3.4.10, 3.5.3, 3.6.0     0 4   {code}
LOG.info("Setting authorizedID: " + userNameBuilder);
ac.setAuthorizedID(userNameBuilder.toString());
} catch (IOException e) {
LOG.error("Failed to set name based on Kerberos authentication rules.");
}
{code}
On one cluster, we saw the following:
{code}
2016-10-04 02:18:16,484 - ERROR [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:SaslServerCallbackHandler@137] - Failed to set name based on Kerberos authentication rules.
{code}
It would be helpful if the log contains information about the IOException.
security 9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
3 years, 22 weeks, 3 days ago 0|i34exr:
ZooKeeper ZOOKEEPER-2605

Snapshot generation fills up disk space due to high volume of requests.

Bug Resolved Minor Not A Problem Unassigned Joe Wang Joe Wang 29/Sep/16 14:55   29/Sep/16 15:20 29/Sep/16 15:20 3.4.5       0 2   Not sure if it's a bug, or just a consequence of a design decision.

Recently we had an issue where faulty clients were issuing create requests at an abnormally high rate, which caused zookeeper to generate more snapshots than our cron job could clean up. This filled up the disk on our zookeeper hosts and brought the cluster down.

Is there a reason why Zookeeper uses a write-ahead log instead only flushing successful transactions to disk? If only successful transactions are flushed and counted towards snapCount, then even if a client is spamming requests to create a node that already exists, it wouldn't cause a flood of snapshots to be persisted to disk.
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 25 weeks ago 0|i349mf:
ZooKeeper ZOOKEEPER-2604

Temporary node has not been deleted

Bug Open Major Unresolved Unassigned sunqb sunqb 27/Sep/16 07:23   14/Mar/18 03:22   3.4.6 3.4.6     0 3   linux centos , jdk7 i use zkclient to connect zookeeper-server , but sometime when i close my zkclient ,the Temporary node can't be deleted。i have search this bug,and i fixed it by delete the data dir 9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
2 years, 1 week, 1 day ago 0|i344s7:
ZooKeeper ZOOKEEPER-2603

Init script restart command broken on Debian Jessie

Bug Open Minor Unresolved Unassigned Mateusz Moneta Mateusz Moneta 26/Sep/16 06:48   26/Sep/16 06:48   3.4.6, 3.4.9       0 2   Hello,

When you try to restart zookeeper via {{service zookeepr restart}} it ends in state:
{noformat}
root@m1:/etc/systemd# service zookeeper status
● zookeeper.service - LSB: Apache ZooKeeper server
Loaded: loaded (/etc/init.d/zookeeper)
Active: active (exited) since Mon 2016-09-26 10:38:47 UTC; 58s ago
Process: 55495 ExecStop=/etc/init.d/zookeeper stop (code=exited, status=0/SUCCESS)
Process: 55504 ExecStart=/etc/init.d/zookeeper start (code=exited, status=0/SUCCESS)
{noformat}

After that {{service zookeeper start}} won't work. Only way is to do {{service zookeeper stop}} and then {{start}} or {{service zookeepr --full-restart}} (which does basically the same).
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 25 weeks, 3 days ago 0|i342kv:
ZooKeeper ZOOKEEPER-2600

dangling ephemerals on overloaded server with local sessions

Bug Resolved Major Cannot Reproduce Unassigned Benjamin Reed Benjamin Reed 22/Sep/16 17:38   24/Sep/16 11:46 24/Sep/16 05:36     quorum   0 4   we had the following strange production bug:

there was an ephemeral znode for a session that was no longer active. it happened even in the absence of failures.

we are running with local sessions enabled and slightly different logic than the open source zookeeper, but code inspection shows that the problem is also in open source.

the triggering condition was server overload. we had a traffic burst and it we were having commit latencies of over 30 seconds.

after digging through logs/code we realized from the logs that the create session txn for the ephemeral node started (in the PrepRequestProcessor) at 11:23:04 and committed at 11:23:38 (the "Adding global session" is output in the commit processor). it took 34 seconds to commit the createSession, during that time the session expired. due to delays it appears that the interleave was as follows:

1) create session hits prep request processor and create session txn generated 11:23:04
2) time passes as the create session is going through zab
3) the session expires, close session is generated, and close session txn generated 11:23:23
4) the create session gets committed and the session gets re-added to the sessionTracker 11:23:38
5) the create ephemeral node hits prep request processor and a create txn generated 11:23:40
6) the close session gets committed (all ephemeral nodes for the session are deleted) and the session is deleted from sessionTracker
7) the create ephemeral node gets committed

the root cause seems to be that the gobal sessions are managed by both the PrepRequestProcessor and the CommitProcessor. also with the local session upgrading we can have changes in flight before our sessions commits. i think there are probably two places to fix:

1) changes to session tracker should not happen in prep request processor.
2) we should not have requests in flight while create session is in process. there are two options to prevent this:
a) when a create session is generated in makeUpgradeRequest, we need to start queuing the requests from the clients and only submit them once the create session is committed
b) the client should explicitly detect that it needs to change from local session to global session and explicitly open a global session and get the commit before it sends an ephemeral create request

option 2a) is a more transparent fix, but architecturally and in the long term i think 2b) might be better.
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 25 weeks, 5 days ago 0|i33z5r:
ZooKeeper ZOOKEEPER-2599

Quorum with 3 nodes, stop 2 nodes and let one running, install and configure new quorum where one node details is common but now has configuration of new quorum. common node getting synced the configuration with previous quorum

Bug Open Critical Unresolved Unassigned Rakesh Kumar Singh Rakesh Kumar Singh 22/Sep/16 10:36   22/Sep/16 10:36   3.5.1       0 3   Start an quorum with 3 (let say A, B, C) zookeepers, stop 2 zookeepers and let one running, install and configure new quorum (A, A2, A3, A4, A5) where A is common but now has configuration of new quorum. When start A, it getting synced the configuration with previous quorum

Steps to reproduce:-
1. Configure and start quorum of 3 nodes (A, B, C) -> 1st quorum
2. stop 2 nodes and let running 3rd node (say C)
3. Create new quorum of 5 nodes (A, A2, A3, A4, A5) where A has same IP and port which was used in 1st quorum but A's configuration is as per new quorum (where details of A, A2, A3, A4, A5) are present and not B & C.
4. Now start 2nd quorum. Here A's dynamic configuration is getting changed according to 1st quorum

Problems:-
1. Now A node is neither syncing all data with 1st quorum nor with 2nd quorum
2. Big security flaw and the whole quorum can be screwed
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 26 weeks ago 0|i33yev:
ZooKeeper ZOOKEEPER-2598

Data Inconsistency after power off/on of some nodes

Bug Open Major Unresolved Unassigned Srinivas Neginhal Srinivas Neginhal 21/Sep/16 14:01   17/Oct/16 06:46   3.5.1   quorum   0 4   ZK is running in a docker container on a Ubuntu 14.04 VM Steps to reproduce:
1. Create a three node cluster: Node1, Node2 and Node3.
Each node is a VM that runs:
1. ZK in a docker container
2. Two clients, A and B that use ZK for group membership and leader election. The clients create sequential ephemeral nodes when they come up.
2. The three ZK's running in the containers form an ensemble.
3. Power off/on Node 2 and Node 3 in a loop
4. After a few times, the ephemeral nodes seen by the three nodes are different.

Here is the output of some four letter commands with the ensemble in the state:

1. conf:

ZK 1:

# echo conf| nc 10.0.0.1 1300
clientPort=1300
secureClientPort=-1
dataDir=/moot/persistentStore/zkWorkspace/version-2
dataDirSize=67293721
dataLogDir=/moot/persistentStore/zkWorkspace/version-2
dataLogSize=67293721
tickTime=2000
maxClientCnxns=60
minSessionTimeout=4000
maxSessionTimeout=40000
serverId=1
initLimit=100
syncLimit=20
electionAlg=3
electionPort=1200
quorumPort=1100
peerType=0
membership:
server.1=10.0.0.1:1100:1200:participant;10.0.0.1:1300;8e64c644-d0fa-414f-bab2-3c8c80364410
server.2=10.0.0.2:1100:1200:participant;10.0.0.2:1300;38bf19b8-d4cb-4dac-b328-7bbf0ee1e2c4
server.3=10.0.0.3:1100:1200:participant;10.0.0.3:1300;e1415d59-e857-43e6-ba9b-01daeb31a434


ZK 2:

# echo conf| nc 10.0.0.2 1300
clientPort=1300
secureClientPort=-1
dataDir=/moot/persistentStore/zkWorkspace/version-2
dataDirSize=1409480873
dataLogDir=/moot/persistentStore/zkWorkspace/version-2
dataLogSize=1409480873
tickTime=2000
maxClientCnxns=60
minSessionTimeout=4000
maxSessionTimeout=40000
serverId=2
initLimit=100
syncLimit=20
electionAlg=3
electionPort=1200
quorumPort=1100
peerType=0
membership:
server.1=10.0.0.1:1100:1200:participant;10.0.0.1:1300;8e64c644-d0fa-414f-bab2-3c8c80364410
server.2=10.0.0.2:1100:1200:participant;10.0.0.2:1300;38bf19b8-d4cb-4dac-b328-7bbf0ee1e2c4
server.3=10.0.0.3:1100:1200:participant;10.0.0.3:1300;e1415d59-e857-43e6-ba9b-01daeb31a434

ZK 3:

# echo conf| nc 10.0.0.3 1300
clientPort=1300
secureClientPort=-1
dataDir=/moot/persistentStore/zkWorkspace/version-2
dataDirSize=1409505467
dataLogDir=/moot/persistentStore/zkWorkspace/version-2
dataLogSize=1409505467
tickTime=2000
maxClientCnxns=60
minSessionTimeout=4000
maxSessionTimeout=40000
serverId=3
initLimit=100
syncLimit=20
electionAlg=3
electionPort=1200
quorumPort=1100
peerType=0
membership:
server.1=10.0.0.1:1100:1200:participant;10.0.0.1:1300;8e64c644-d0fa-414f-bab2-3c8c80364410
server.2=10.0.0.2:1100:1200:participant;10.0.0.2:1300;38bf19b8-d4cb-4dac-b328-7bbf0ee1e2c4
server.3=10.0.0.3:1100:1200:participant;10.0.0.3:1300;e1415d59-e857-43e6-ba9b-01daeb31a434


2. mntr:

ZK 1:

# echo mntr| nc 10.0.0.1 1300
zk_version 3.5.1-alpha--1, built on 09/07/2016 00:34 GMT
zk_avg_latency 0
zk_max_latency 471
zk_min_latency 0
zk_packets_received 32556
zk_packets_sent 32564
zk_num_alive_connections 7
zk_outstanding_requests 0
zk_server_state leader
zk_znode_count 58
zk_watch_count 51
zk_ephemerals_count 5
zk_approximate_data_size 5251
zk_open_file_descriptor_count 52
zk_max_file_descriptor_count 1048576
zk_followers 2
zk_synced_followers 2
zk_pending_syncs 0

ZK 2:

# echo mntr| nc 10.0.0.2 1300
zk_version 3.5.1-alpha--1, built on 09/07/2016 00:34 GMT
zk_avg_latency 1
zk_max_latency 227
zk_min_latency 0
zk_packets_received 30905
zk_packets_sent 30936
zk_num_alive_connections 6
zk_outstanding_requests 0
zk_server_state follower
zk_znode_count 58
zk_watch_count 82
zk_ephemerals_count 5
zk_approximate_data_size 5251
zk_open_file_descriptor_count 49
zk_max_file_descriptor_count 1048576

ZK 3:

# echo mntr| nc 10.0.0.3 1300
zk_version 3.5.1-alpha--1, built on 09/07/2016 00:34 GMT
zk_avg_latency 4
zk_max_latency 590
zk_min_latency 0
zk_packets_received 6192
zk_packets_sent 6191
zk_num_alive_connections 2
zk_outstanding_requests 0
zk_server_state follower
zk_znode_count 64
zk_watch_count 17
zk_ephemerals_count 11
zk_approximate_data_size 5806
zk_open_file_descriptor_count 45
zk_max_file_descriptor_count 1048576


3. dump showing the inconsistency:

ZK 1:

# echo dump| nc 10.0.0.1 1300
SessionTracker dump:
Session Sets (17)/(12):
0 expire at Tue Sep 20 18:22:35 UTC 2016:
0 expire at Tue Sep 20 18:22:37 UTC 2016:
0 expire at Tue Sep 20 18:22:39 UTC 2016:
0 expire at Tue Sep 20 18:22:41 UTC 2016:
0 expire at Tue Sep 20 18:22:43 UTC 2016:
0 expire at Tue Sep 20 18:22:45 UTC 2016:
0 expire at Tue Sep 20 18:22:49 UTC 2016:
0 expire at Tue Sep 20 18:22:51 UTC 2016:
0 expire at Tue Sep 20 18:22:53 UTC 2016:
0 expire at Tue Sep 20 18:22:55 UTC 2016:
0 expire at Tue Sep 20 18:22:57 UTC 2016:
4 expire at Tue Sep 20 18:22:59 UTC 2016:
0x100061435f7000d
0x10000d9e4460004
0x100061435f70002
0x10000d9e4460003
4 expire at Tue Sep 20 18:23:03 UTC 2016:
0x2000001141a0002
0x2000001141a0000
0x2000001141a0005
0x100061435f70010
1 expire at Tue Sep 20 18:23:07 UTC 2016:
0x2000001141a0001
1 expire at Tue Sep 20 18:23:09 UTC 2016:
0x100061435f70000
1 expire at Tue Sep 20 18:23:11 UTC 2016:
0x2000001141a000f
1 expire at Tue Sep 20 18:23:13 UTC 2016:
0x300000188c30001
ephemeral nodes dump:
Sessions with Ephemerals (5):
0x100061435f70000:
/moot/gmle/ServiceDirectory/ActiveNodes/member0000000064
0x2000001141a000f:
/moot/gmle/ServiceDirectory/ActiveNodes/member0000000066
0x2000001141a0001:
/moot/gmle/ServiceDirectory/ActiveNodes/member0000000065
0x2000001141a0000:
/moot/gmle/ActiveControllerCluster/member0000000065
0x2000001141a0005:
/moot/gmle/ActiveControllerCluster/member0000000066
Connections dump:
Connections Sets (5)/(10):
0 expire at Tue Sep 20 18:22:35 UTC 2016:
1 expire at Tue Sep 20 18:22:45 UTC 2016:
ip: /10.0.0.1:45591 sessionId: 0x0
0 expire at Tue Sep 20 18:22:55 UTC 2016:
5 expire at Tue Sep 20 18:23:05 UTC 2016:
ip: /10.0.0.3:34734 sessionId: 0x100061435f7000d
ip: /10.0.0.1:42963 sessionId: 0x10000d9e4460003
ip: /10.0.0.3:34739 sessionId: 0x100061435f70010
ip: /10.0.0.2:45750 sessionId: 0x100061435f70002
ip: /10.0.0.1:42961 sessionId: 0x10000d9e4460004
1 expire at Tue Sep 20 18:23:15 UTC 2016:
ip: /10.0.0.1:42964 sessionId: 0x100061435f70000

ZK 2:

# echo dump| nc 10.0.0.2 1300
SessionTracker dump:
Global Sessions(13):
0x10000d9e4460003 30000ms
0x10000d9e4460004 30000ms
0x100061435f70000 40000ms
0x100061435f70002 30000ms
0x100061435f7000d 30000ms
0x100061435f70010 30000ms
0x100061435f70584 4000ms
0x2000001141a0000 40000ms
0x2000001141a0001 40000ms
0x2000001141a0002 30000ms
0x2000001141a0005 40000ms
0x2000001141a000f 40000ms
0x300000188c30001 40000ms
ephemeral nodes dump:
Sessions with Ephemerals (5):
0x100061435f70000:
/moot/gmle/ServiceDirectory/ActiveNodes/member0000000064
0x2000001141a000f:
/moot/gmle/ServiceDirectory/ActiveNodes/member0000000066
0x2000001141a0001:
/moot/gmle/ServiceDirectory/ActiveNodes/member0000000065
0x2000001141a0000:
/moot/gmle/ActiveControllerCluster/member0000000065
0x2000001141a0005:
/moot/gmle/ActiveControllerCluster/member0000000066
Connections dump:
Connections Sets (4)/(6):
0 expire at Tue Sep 20 18:25:13 UTC 2016:
1 expire at Tue Sep 20 18:25:23 UTC 2016:
ip: /10.0.0.2:38021 sessionId: 0x0
1 expire at Tue Sep 20 18:25:33 UTC 2016:
ip: /10.0.0.2:35422 sessionId: 0x2000001141a0002
4 expire at Tue Sep 20 18:25:43 UTC 2016:
ip: /10.0.0.2:35419 sessionId: 0x2000001141a0001
ip: /10.0.0.1:59025 sessionId: 0x2000001141a0000
ip: /10.0.0.2:35427 sessionId: 0x2000001141a0005
ip: /10.0.0.3:56967 sessionId: 0x2000001141a000f

ZK 3:

# echo dump| nc 10.0.0.3 1300
SessionTracker dump:
Global Sessions(23):
0x10000d9e4460003 30000ms
0x10000d9e4460004 30000ms
0x100055a50b00001 30000ms
0x100055a50b00003 40000ms
0x100055a50b0000c 40000ms
0x100061435f70000 40000ms
0x100061435f70002 30000ms
0x100061435f7000d 30000ms
0x100061435f70010 30000ms
0x100061435f70585 4000ms
0x2000001141a0000 40000ms
0x2000001141a0001 40000ms
0x2000001141a0002 30000ms
0x2000001141a0005 40000ms
0x2000001141a000f 40000ms
0x200000130750000 40000ms
0x200000130750001 40000ms
0x200000130750002 30000ms
0x200000130750004 40000ms
0x20000013075000d 30000ms
0x3000000e4860000 30000ms
0x3000000e4860002 40000ms
0x300000188c30001 40000ms
ephemeral nodes dump:
Sessions with Ephemerals (11):
0x100061435f70000:
/moot/gmle/ServiceDirectory/ActiveNodes/member0000000064
0x3000000e4860002:
/moot/gmle/ActiveControllerCluster/member0000000027
0x100055a50b0000c:
/moot/gmle/ServiceDirectory/ActiveNodes/member0000000027
0x100055a50b00003:
/moot/gmle/ActiveControllerCluster/member0000000025
0x200000130750004:
/moot/gmle/ActiveControllerCluster/member0000000026
0x200000130750000:
/moot/gmle/ServiceDirectory/ActiveNodes/member0000000026
0x2000001141a000f:
/moot/gmle/ServiceDirectory/ActiveNodes/member0000000066
0x200000130750001:
/moot/gmle/ServiceDirectory/ActiveNodes/member0000000025
0x2000001141a0001:
/moot/gmle/ServiceDirectory/ActiveNodes/member0000000065
0x2000001141a0000:
/moot/gmle/ActiveControllerCluster/member0000000065
0x2000001141a0005:
/moot/gmle/ActiveControllerCluster/member0000000066
Connections dump:
Connections Sets (4)/(2):
0 expire at Tue Sep 20 18:25:40 UTC 2016:
1 expire at Tue Sep 20 18:25:50 UTC 2016:
ip: /10.0.0.3:52784 sessionId: 0x0
0 expire at Tue Sep 20 18:26:10 UTC 2016:
1 expire at Tue Sep 20 18:26:20 UTC 2016:
ip: /10.0.0.3:50222 sessionId: 0x300000188c30001
9223372036854775807 No Perforce job exists for this issue. 3 9223372036854775807
3 years, 22 weeks, 3 days ago 0|i33wsv:
ZooKeeper ZOOKEEPER-2597

Add script to merge PR from Apache git repo to Github

Improvement Resolved Minor Fixed Edward Ribeiro Edward Ribeiro Edward Ribeiro 21/Sep/16 09:01   15/Feb/17 21:24 25/Oct/16 22:15         0 6   A port of kafka-merge-pr.py to workon on ZooKeeper repo. 9223372036854775807 No Perforce job exists for this issue. 2 9223372036854775807
3 years, 21 weeks, 1 day ago 0|i33w6f:
ZooKeeper ZOOKEEPER-2596

Zookeeper.c - gethostname drops subdomain returning only partial FQDN

Bug Resolved Minor Invalid Unassigned Scott Thompson Scott Thompson 20/Sep/16 10:10   21/Sep/16 04:28 20/Sep/16 17:11     c client   0 2   RedHat Enterprise Server 7.2 Nodes fail to connect when a sub-domain is present in the FQDN.

The sub-domain is dropped from the hostname string when calling gethostname in zookeeper.c.

machine.sub.domain.com
becomes
machine.domain.com

#ifdef HAVE_GETHOSTNAME
gethostname(buf, sizeof(buf));
LOG_INFO(LOGCALLBACK(zh), "Client environment:host.name=%s", buf);
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 26 weeks, 1 day ago 0|i33ud3:
ZooKeeper ZOOKEEPER-2595

znode created with acl enabled on it can be deleted by any unauthorised user, when it has no child znodes

Bug Open Major Unresolved Unassigned Neha Bathra Neha Bathra 20/Sep/16 09:13   20/Sep/16 09:15           0 2   user1 sets ACL on one znode for user 2
example :
create /xyz data sasl:user2/xyz@XYZ.COM:cdr


now user3 can login to zkCli and delete /xyz if it has no children nodes, even when it does not have access
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 26 weeks, 2 days ago 0|i33uaf:
ZooKeeper ZOOKEEPER-2594

Use TLS for downloading artifacts during build

Improvement Closed Blocker Fixed Olaf Flebbe Olaf Flebbe Olaf Flebbe 19/Sep/16 15:06   31/Mar/17 05:01 05/Oct/16 12:33 3.4.9, 3.5.2 3.4.10, 3.5.3, 3.6.0 build   0 2   Zookeeper builds are downloading dependencies using the insecure http:// protocol.

An outdated java.net repository can be removed now, since its content is now on maven.org.

The https://repo2.maven.org cannot be used, since its certificate is invalid. Use repo1.maven.org instead (IMHO this is intentional).

Appended you'll find a proposed patch (against git head) to fix these issues, for a starter.
security 9223372036854775807 No Perforce job exists for this issue. 5 9223372036854775807
3 years, 24 weeks, 1 day ago
Reviewed
0|i33t8f:
ZooKeeper ZOOKEEPER-2593

Enforce the quota limit

New Feature Open Major Unresolved Mohammad Arshad Mohammad Arshad Mohammad Arshad 19/Sep/16 11:40   08/May/19 02:13       java client, server   0 4   Currently in ZooKeeper when quota limit exceeds, a warning is logged. There are many user scenarios where it is desired to throw exception in case quota limits exceed.
We should make it configurable whether to throw exception or just log the warning when quota limits exceed.
*Implementation:*
add new properties
{code}
enforce.number.quota
enforce.byte.quota
{code}
add new error codes
{code}
KeeperException.Code.NUMBERQUOTAEXCEED
KeeperException.Code.BYTEQUOTAEXCEED
{code}
add new exception
{code}
KeeperException.NumberQuotaExceedException
KeeperException.ByteQuotaExceedException
{code}

*Basic Scenarios:*
# If enforce.number.quota=true and number quota exceed, then server should send NUMBERQUOTAEXCEED error code and client should throw NumberQuotaExceedException
# If enforce.byte.quota=true and byte quota exceed, then server should send BYTEQUOTAEXCEED error code and client should throw ByteQuotaExceedException

*Impacted APIs:*
create
setData
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 26 weeks ago 0|i33swn:
ZooKeeper ZOOKEEPER-2592

Zookeeper is not recoverable once running system( machine on which zookeeper is running) is out of space

Bug Open Critical Unresolved Unassigned Rakesh Kumar Singh Rakesh Kumar Singh 19/Sep/16 11:01   21/Nov/18 20:04   3.5.1, 3.5.2   server   1 7   Zookeeper is not recoverable once running system( machine on which zookeeper is running) is out of space
Steps to reproduce:-
1. Install zookeeper on standalone mode and start zookeeper
2. Make the machine physical memory full
3. Connect through client to zookeeper and trying create some znodes with some data.
4. After sometime creating further znode will not happened as complete memory is occupied
5. Now start creating space in that machine
6. Again connect through a client. Connection is fine. Now try to execute any command like "ls / " it fails even though now space is more than 11gb

Client log:-
BLR1000007042:/home/Rakesh/Zookeeper/18_Aug/cluster/zookeeper-3.5.1-alpha/bin # df -h
Filesystem Size Used Avail Use% Mounted on
/dev/xvda2 36G 24G 11G 70% /
udev 1.9G 116K 1.9G 1% /dev
tmpfs 1.9G 0 1.9G 0% /dev/shm
BLR1000007042:/home/Rakesh/Zookeeper/18_Aug/cluster/zookeeper-3.5.1-alpha/bin # ./zkCli.sh
Connecting to localhost:2181
2016-09-19 22:50:20,227 [myid:] - INFO [main:Environment@109] - Client environment:zookeeper.version=3.5.1-alpha--1, built on 08/18/2016 08:20 GMT
2016-09-19 22:50:20,231 [myid:] - INFO [main:Environment@109] - Client environment:host.name=BLR1000007042
2016-09-19 22:50:20,231 [myid:] - INFO [main:Environment@109] - Client environment:java.version=1.7.0_79
2016-09-19 22:50:20,234 [myid:] - INFO [main:Environment@109] - Client environment:java.vendor=Oracle Corporation
2016-09-19 22:50:20,234 [myid:] - INFO [main:Environment@109] - Client environment:java.home=/usr/java/jdk1.7.0_79/jre
2016-09-19 22:50:20,234 [myid:] - INFO [main:Environment@109] - Client environment:java.class.path=/home/Rakesh/Zookeeper/18_Aug/cluster/zookeeper-3.5.1-alpha/bin/../build/classes:/home/Rakesh/Zookeeper/18_Aug/cluster/zookeeper-3.5.1-alpha/bin/../build/lib/*.jar:/home/Rakesh/Zookeeper/18_Aug/cluster/zookeeper-3.5.1-alpha/bin/../lib/slf4j-log4j12-1.7.5.jar:/home/Rakesh/Zookeeper/18_Aug/cluster/zookeeper-3.5.1-alpha/bin/../lib/slf4j-api-1.7.5.jar:/home/Rakesh/Zookeeper/18_Aug/cluster/zookeeper-3.5.1-alpha/bin/../lib/servlet-api-2.5-20081211.jar:/home/Rakesh/Zookeeper/18_Aug/cluster/zookeeper-3.5.1-alpha/bin/../lib/netty-3.7.0.Final.jar:/home/Rakesh/Zookeeper/18_Aug/cluster/zookeeper-3.5.1-alpha/bin/../lib/log4j-1.2.16.jar:/home/Rakesh/Zookeeper/18_Aug/cluster/zookeeper-3.5.1-alpha/bin/../lib/jline-2.11.jar:/home/Rakesh/Zookeeper/18_Aug/cluster/zookeeper-3.5.1-alpha/bin/../lib/jetty-util-6.1.26.jar:/home/Rakesh/Zookeeper/18_Aug/cluster/zookeeper-3.5.1-alpha/bin/../lib/jetty-6.1.26.jar:/home/Rakesh/Zookeeper/18_Aug/cluster/zookeeper-3.5.1-alpha/bin/../lib/javacc.jar:/home/Rakesh/Zookeeper/18_Aug/cluster/zookeeper-3.5.1-alpha/bin/../lib/jackson-mapper-asl-1.9.11.jar:/home/Rakesh/Zookeeper/18_Aug/cluster/zookeeper-3.5.1-alpha/bin/../lib/jackson-core-asl-1.9.11.jar:/home/Rakesh/Zookeeper/18_Aug/cluster/zookeeper-3.5.1-alpha/bin/../lib/commons-cli-1.2.jar:/home/Rakesh/Zookeeper/18_Aug/cluster/zookeeper-3.5.1-alpha/bin/../lib/ant-eclipse-1.0-jvm1.2.jar:/home/Rakesh/Zookeeper/18_Aug/cluster/zookeeper-3.5.1-alpha/bin/../zookeeper-3.5.1-alpha.jar:/home/Rakesh/Zookeeper/18_Aug/cluster/zookeeper-3.5.1-alpha/bin/../src/java/lib/ant-eclipse-1.0-jvm1.2.jar:/home/Rakesh/Zookeeper/18_Aug/cluster/zookeeper-3.5.1-alpha/bin/../conf:/usr/java/jdk1.7.0_79/lib
2016-09-19 22:50:20,234 [myid:] - INFO [main:Environment@109] - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2016-09-19 22:50:20,234 [myid:] - INFO [main:Environment@109] - Client environment:java.io.tmpdir=/tmp
2016-09-19 22:50:20,234 [myid:] - INFO [main:Environment@109] - Client environment:java.compiler=<NA>
2016-09-19 22:50:20,235 [myid:] - INFO [main:Environment@109] - Client environment:os.name=Linux
2016-09-19 22:50:20,235 [myid:] - INFO [main:Environment@109] - Client environment:os.arch=amd64
2016-09-19 22:50:20,235 [myid:] - INFO [main:Environment@109] - Client environment:os.version=3.0.76-0.11-default
2016-09-19 22:50:20,235 [myid:] - INFO [main:Environment@109] - Client environment:user.name=root
2016-09-19 22:50:20,235 [myid:] - INFO [main:Environment@109] - Client environment:user.home=/root
2016-09-19 22:50:20,235 [myid:] - INFO [main:Environment@109] - Client environment:user.dir=/home/Rakesh/Zookeeper/18_Aug/cluster/zookeeper-3.5.1-alpha/bin
2016-09-19 22:50:20,235 [myid:] - INFO [main:Environment@109] - Client environment:os.memory.free=52MB
2016-09-19 22:50:20,237 [myid:] - INFO [main:Environment@109] - Client environment:os.memory.max=227MB
2016-09-19 22:50:20,238 [myid:] - INFO [main:Environment@109] - Client environment:os.memory.total=57MB
2016-09-19 22:50:20,241 [myid:] - INFO [main:ZooKeeper@716] - Initiating client connection, connectString=localhost:2181 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@3865db85
Welcome to ZooKeeper!
2016-09-19 22:50:20,264 [myid:] - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread@1138] - Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
2016-09-19 22:50:20,270 [myid:] - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread@980] - Socket connection established, initiating session, client: /127.0.0.1:47801, server: localhost/127.0.0.1:2181
JLine support is enabled
[INFO] Unable to bind key for unsupported operation: backward-delete-word
[INFO] Unable to bind key for unsupported operation: backward-delete-word
[INFO] Unable to bind key for unsupported operation: down-history
[INFO] Unable to bind key for unsupported operation: up-history
[INFO] Unable to bind key for unsupported operation: up-history
[INFO] Unable to bind key for unsupported operation: down-history
[INFO] Unable to bind key for unsupported operation: up-history
[INFO] Unable to bind key for unsupported operation: down-history
[INFO] Unable to bind key for unsupported operation: up-history
[INFO] Unable to bind key for unsupported operation: down-history
[INFO] Unable to bind key for unsupported operation: up-history
[INFO] Unable to bind key for unsupported operation: down-history
[zk: localhost:2181(CONNECTING) 0] ls /
2016-09-19 22:50:35,280 [myid:] - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread@1251] - Client session timed out, have not heard from server in 15011ms for sessionid 0x0, closing socket connection and attempting reconnect
Exception in thread "main" org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /
at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:2255)
at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:2283)
at org.apache.zookeeper.cli.LsCommand.exec(LsCommand.java:93)
at org.apache.zookeeper.ZooKeeperMain.processZKCmd(ZooKeeperMain.java:674)
at org.apache.zookeeper.ZooKeeperMain.processCmd(ZooKeeperMain.java:577)
at org.apache.zookeeper.ZooKeeperMain.executeLine(ZooKeeperMain.java:360)
at org.apache.zookeeper.ZooKeeperMain.run(ZooKeeperMain.java:320)
at org.apache.zookeeper.ZooKeeperMain.main(ZooKeeperMain.java:280)
BLR1000007042:/home/Rakesh/Zookeeper/18_Aug/cluster/zookeeper-3.5.1-alpha/bin #


-{color:blue} Server log

2016-09-19 22:34:13,380 [myid:] - INFO [main:QuorumPeerConfig@114] - Reading configuration from: /home/Rakesh/Zookeeper/18_Aug/cluster/zookeeper-3.5.1-alpha/bin/../conf/zoo.cfg
2016-09-19 22:34:13,386 [myid:] - INFO [main:QuorumPeerConfig@316] - clientPortAddress is 0.0.0.0/0.0.0.0:2181
2016-09-19 22:34:13,386 [myid:] - INFO [main:QuorumPeerConfig@320] - secureClientPort is not set
2016-09-19 22:34:13,389 [myid:] - INFO [main:DatadirCleanupManager@78] - autopurge.snapRetainCount set to 3
2016-09-19 22:34:13,389 [myid:] - INFO [main:DatadirCleanupManager@79] - autopurge.purgeInterval set to 0
2016-09-19 22:34:13,390 [myid:] - INFO [main:DatadirCleanupManager@101] - Purge task is not scheduled.
2016-09-19 22:34:13,390 [myid:] - WARN [main:QuorumPeerMain@122] - Either no config or no quorum defined in config, running in standalone mode
2016-09-19 22:34:13,402 [myid:] - INFO [main:QuorumPeerConfig@114] - Reading configuration from: /home/Rakesh/Zookeeper/18_Aug/cluster/zookeeper-3.5.1-alpha/bin/../conf/zoo.cfg
2016-09-19 22:34:13,402 [myid:] - INFO [main:QuorumPeerConfig@316] - clientPortAddress is 0.0.0.0/0.0.0.0:2181
2016-09-19 22:34:13,402 [myid:] - INFO [main:QuorumPeerConfig@320] - secureClientPort is not set
2016-09-19 22:34:13,403 [myid:] - INFO [main:ZooKeeperServerMain@113] - Starting server
2016-09-19 22:34:13,416 [myid:] - INFO [main:Environment@109] - Server environment:zookeeper.version=3.5.1-alpha--1, built on 08/18/2016 08:20 GMT
2016-09-19 22:34:13,416 [myid:] - INFO [main:Environment@109] - Server environment:host.name=BLR1000007042
2016-09-19 22:34:13,416 [myid:] - INFO [main:Environment@109] - Server environment:java.version=1.7.0_79
2016-09-19 22:34:13,417 [myid:] - INFO [main:Environment@109] - Server environment:java.vendor=Oracle Corporation
2016-09-19 22:34:13,417 [myid:] - INFO [main:Environment@109] - Server environment:java.home=/usr/java/jdk1.7.0_79/jre
2016-09-19 22:34:13,419 [myid:] - INFO [main:Environment@109] - Server environment:java.class.path=/home/Rakesh/Zookeeper/18_Aug/cluster/zookeeper-3.5.1-alpha/bin/../build/classes:/home/Rakesh/Zookeeper/18_Aug/cluster/zookeeper-3.5.1-alpha/bin/../build/lib/*.jar:/home/Rakesh/Zookeeper/18_Aug/cluster/zookeeper-3.5.1-alpha/bin/../lib/slf4j-log4j12-1.7.5.jar:/home/Rakesh/Zookeeper/18_Aug/cluster/zookeeper-3.5.1-alpha/bin/../lib/slf4j-api-1.7.5.jar:/home/Rakesh/Zookeeper/18_Aug/cluster/zookeeper-3.5.1-alpha/bin/../lib/servlet-api-2.5-20081211.jar:/home/Rakesh/Zookeeper/18_Aug/cluster/zookeeper-3.5.1-alpha/bin/../lib/netty-3.7.0.Final.jar:/home/Rakesh/Zookeeper/18_Aug/cluster/zookeeper-3.5.1-alpha/bin/../lib/log4j-1.2.16.jar:/home/Rakesh/Zookeeper/18_Aug/cluster/zookeeper-3.5.1-alpha/bin/../lib/jline-2.11.jar:/home/Rakesh/Zookeeper/18_Aug/cluster/zookeeper-3.5.1-alpha/bin/../lib/jetty-util-6.1.26.jar:/home/Rakesh/Zookeeper/18_Aug/cluster/zookeeper-3.5.1-alpha/bin/../lib/jetty-6.1.26.jar:/home/Rakesh/Zookeeper/18_Aug/cluster/zookeeper-3.5.1-alpha/bin/../lib/javacc.jar:/home/Rakesh/Zookeeper/18_Aug/cluster/zookeeper-3.5.1-alpha/bin/../lib/jackson-mapper-asl-1.9.11.jar:/home/Rakesh/Zookeeper/18_Aug/cluster/zookeeper-3.5.1-alpha/bin/../lib/jackson-core-asl-1.9.11.jar:/home/Rakesh/Zookeeper/18_Aug/cluster/zookeeper-3.5.1-alpha/bin/../lib/commons-cli-1.2.jar:/home/Rakesh/Zookeeper/18_Aug/cluster/zookeeper-3.5.1-alpha/bin/../lib/ant-eclipse-1.0-jvm1.2.jar:/home/Rakesh/Zookeeper/18_Aug/cluster/zookeeper-3.5.1-alpha/bin/../zookeeper-3.5.1-alpha.jar:/home/Rakesh/Zookeeper/18_Aug/cluster/zookeeper-3.5.1-alpha/bin/../src/java/lib/ant-eclipse-1.0-jvm1.2.jar:/home/Rakesh/Zookeeper/18_Aug/cluster/zookeeper-3.5.1-alpha/bin/../conf:/usr/java/jdk1.7.0_79/lib
2016-09-19 22:34:13,420 [myid:] - INFO [main:Environment@109] - Server environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2016-09-19 22:34:13,420 [myid:] - INFO [main:Environment@109] - Server environment:java.io.tmpdir=/tmp
2016-09-19 22:34:13,420 [myid:] - INFO [main:Environment@109] - Server environment:java.compiler=<NA>
2016-09-19 22:34:13,420 [myid:] - INFO [main:Environment@109] - Server environment:os.name=Linux
2016-09-19 22:34:13,420 [myid:] - INFO [main:Environment@109] - Server environment:os.arch=amd64
2016-09-19 22:34:13,421 [myid:] - INFO [main:Environment@109] - Server environment:os.version=3.0.76-0.11-default
2016-09-19 22:34:13,421 [myid:] - INFO [main:Environment@109] - Server environment:user.name=root
2016-09-19 22:34:13,421 [myid:] - INFO [main:Environment@109] - Server environment:user.home=/root
2016-09-19 22:34:13,421 [myid:] - INFO [main:Environment@109] - Server environment:user.dir=/home/Rakesh/Zookeeper/18_Aug/cluster/zookeeper-3.5.1-alpha/bin
2016-09-19 22:34:13,421 [myid:] - INFO [main:Environment@109] - Server environment:os.memory.free=51MB
2016-09-19 22:34:13,422 [myid:] - INFO [main:Environment@109] - Server environment:os.memory.max=889MB
2016-09-19 22:34:13,422 [myid:] - INFO [main:Environment@109] - Server environment:os.memory.total=57MB
2016-09-19 22:34:13,424 [myid:] - INFO [main:ZooKeeperServer@858] - minSessionTimeout set to 4000
2016-09-19 22:34:13,424 [myid:] - INFO [main:ZooKeeperServer@867] - maxSessionTimeout set to 40000
2016-09-19 22:34:13,424 [myid:] - INFO [main:ZooKeeperServer@156] - Created server with tickTime 2000 minSessionTimeout 4000 maxSessionTimeout 40000 datadir /home/Rakesh/Zookeeper/18_Aug/cluster/zookeeper-3.5.1-alpha/zoo_log/version-2 snapdir /home/Rakesh/Zookeeper/18_Aug/cluster/zookeeper-3.5.1-alpha/data/version-2
2016-09-19 22:34:13,453 [myid:] - INFO [main:Slf4jLog@67] - Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
2016-09-19 22:34:13,477 [myid:] - INFO [main:Slf4jLog@67] - jetty-6.1.26
2016-09-19 22:34:13,510 [myid:] - INFO [main:Slf4jLog@67] - Started SelectChannelConnector@0.0.0.0:8080
2016-09-19 22:34:13,514 [myid:] - INFO [main:JettyAdminServer@105] - Started AdminServer on address 0.0.0.0, port 8080 and command URL /commands
2016-09-19 22:34:13,521 [myid:] - INFO [main:NIOServerCnxnFactory@673] - Configuring NIO connection handler with 10s sessionless connection timeout, 1 selector thread(s), 8 worker threads, and 64 kB direct buffers.
2016-09-19 22:34:13,523 [myid:] - INFO [main:NIOServerCnxnFactory@686] - binding to port 0.0.0.0/0.0.0.0:2181
2016-09-19 22:34:13,537 [myid:] - INFO [main:FileTxnSnapLog@298] - Snapshotting: 0x0 to /home/Rakesh/Zookeeper/18_Aug/cluster/zookeeper-3.5.1-alpha/data/version-2/snapshot.0
2016-09-19 22:34:13,567 [myid:] - INFO [main:ContainerManager@64] - Using checkIntervalMs=60000 maxPerMinute=10000
2016-09-19 22:35:41,907 [myid:] - INFO [NIOServerCxnFactory.AcceptThread:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /0:0:0:0:0:0:0:1:49485
2016-09-19 22:35:41,917 [myid:] - INFO [NIOWorkerThread-1:ZooKeeperServer@964] - Client attempting to establish new session at /0:0:0:0:0:0:0:1:49485
2016-09-19 22:35:41,919 [myid:] - INFO [SyncThread:0:FileTxnLog@200] - Creating new log file: log.1
2016-09-19 22:35:41,952 [myid:] - INFO [SyncThread:0:ZooKeeperServer@678] - Established session 0x100632436270000 with negotiated timeout 30000 for client /0:0:0:0:0:0:0:1:49485
2016-09-19 22:40:21,211 [myid:] - INFO [NIOServerCxnFactory.AcceptThread:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /10.18.221.194:34892
2016-09-19 22:40:21,218 [myid:] - INFO [NIOWorkerThread-8:ZooKeeperServer@964] - Client attempting to establish new session at /10.18.221.194:34892
2016-09-19 22:40:21,221 [myid:] - INFO [SyncThread:0:ZooKeeperServer@678] - Established session 0x100632436270001 with negotiated timeout 30000 for client /10.18.221.194:34892
2016-09-19 22:40:40,298 [myid:] - INFO [ProcessThread(sid:0 cport:2181)::PrepRequestProcessor@649] - Processed session termination for sessionid: 0x100632436270001
2016-09-19 22:40:40,301 [myid:] - INFO [NIOWorkerThread-3:MBeanRegistry@119] - Unregister MBean [org.apache.ZooKeeperService:name0=StandaloneServer_port2181,name1=Connections,name2=10.18.221.194,name3=0x100632436270001]
2016-09-19 22016-09-19 22:43:47,733 [myid:] - INFO [SyncThread:0:ZooKeeperServer@498] - shutting down
2016-09-19 22:44:39,892 [myid:] - INFO [NIOServerCxnFactory.AcceptThread:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:47796
2016-09-19 22:44:39,898 [myid:] - INFO [NIOWorkerThread-2:ZooKeeperServer@964] - Client attempting to establish new session at /127.0.0.1:47796
2016-09-19 22:45:15,883 [myid:] - INFO [NIOServerCxnFactory.AcceptThread:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /0:0:0:0:0:0:0:1:49493
2016-09-19 22:45:15,890 [myid:] - INFO [NIOWorkerThread-3:ZooKeeperServer@964] - Client attempting to establish new session at /0:0:0:0:0:0:0:1:49493
2016-09-19 22:45:16,000 [myid:] - INFO [ConnnectionExpirer:NIOServerCnxn@606] - Closed socket connection for client /127.0.0.1:47796 which had sessionid 0x100632436270012
2016-09-19 22:45:46,000 [myid:] - INFO [ConnnectionExpirer:NIOServerCnxn@606] - Closed socket connection for client /0:0:0:0:0:0:0:1:49493 which had sessionid 0x100632436270013
2016-09-19 22:47:42,512 [myid:] - INFO [NIOServerCxnFactory.AcceptThread:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /0:0:0:0:0:0:0:1:49494
2016-09-19 22:47:42,519 [myid:] - INFO [NIOWorkerThread-4:ZooKeeperServer@964] - Client attempting to establish new session at /0:0:0:0:0:0:0:1:49494
2016-09-19 22:48:16,001 [myid:] - INFO [ConnnectionExpirer:NIOServerCnxn@606] - Closed socket connection for client /0:0:0:0:0:0:0:1:49494 which had sessionid 0x100632436270014
2016-09-19 22:50:20,268 [myid:] - INFO [NIOServerCxnFactory.AcceptThread:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:47801
2016-09-19 22:50:20,275 [myid:] - INFO [NIOWorkerThread-5:ZooKeeperServer@964] - Client attempting to establish new session at /127.0.0.1:47801
2016-09-19 22:50:56,000 [myid:] - INFO [ConnnectionExpirer:NIOServerCnxn@606] - Closed socket connection for client /127.0.0.1:47801 which had sessionid 0x100632436270015

{color}
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
1 year, 17 weeks ago 0|i33str:
ZooKeeper ZOOKEEPER-2591

The deletion of Container znode doesn't check ACL delete permission

Bug Resolved Major Not A Bug Edward Ribeiro Edward Ribeiro Edward Ribeiro 17/Sep/16 22:16   19/Mar/19 08:45 19/Sep/16 10:41     security, server   0 6   Container nodes check the ACL before creation, but the deletion doesn't check the ACL rights. The code below succeeds even tough we removed ACL access permissions for "/a".

{code}
zk.create("/a", null, Ids.OPEN_ACL_UNSAFE, CreateMode.CONTAINER);
ArrayList<ACL> list = new ArrayList<>();
list.add(new ACL(0, Ids.ANYONE_ID_UNSAFE));
zk.setACL("/", list, -1);

zk.delete("/a", -1);
{code}
container_znode_type 9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
2 years, 36 weeks, 5 days ago 0|i33ref:
ZooKeeper ZOOKEEPER-2590

setACL doesn't affect exists() operation

Bug Open Major Unresolved Unassigned Edward Ribeiro Edward Ribeiro 17/Sep/16 21:51   06/Nov/19 13:54           0 2   As hinted [here|https://github.com/apache/zookeeper/blob/master/src/java/main/org/apache/zookeeper/server/FinalRequestProcessor.java#L298], even if a parent znode path has restricted READ access it's possible to issue an exists() operation on any child znode of that given path.

For example, the snippet below doesn't throw {{NoAuthExceptio}}, even tough it removes ACL rights to "/":

{code}
zk.create("/a", null, Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT);
ArrayList<ACL> acls = new ArrayList<>();
acls.add(new ACL(0, Ids.ANYONE_ID_UNSAFE));

zk.setACL("/", acls, -1);

Stat r = zk.exists("/a", false);
{code}

Also, in the above example, what if the removed READ access for "/a"? Should we allow a call to exists("/a") to succeed even if it returns the znode metadata info?
acl, security 9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
1 year, 34 weeks ago 0|i33rdz:
ZooKeeper ZOOKEEPER-2589

Not able to access znode if IP ACL is set on a znode when zookeeper started in ssl mode

Bug Open Major Unresolved Unassigned Rakesh Kumar Singh Rakesh Kumar Singh 17/Sep/16 02:18   21/Sep/16 01:12   3.5.1       0 3   Not able to access znode if IP ACL is set on a znode when zookeeper started in ssl mode.

Steps to reproduce:-
1. Start zookeeper in SSL (standalone) mode
2. Create a znode
3. set ip ACL and connect the zkCli and try to access, it does not allow.

[zk: localhost:2181(CONNECTED) 3] setAcl /test ip:127.0.0.1:crdwa
[zk: localhost:2181(CONNECTED) 5] quit

>> start the zkCli with 127.0.0.1 and trying access the znode
[zk: 127.0.0.1:2181(CONNECTED) 0] get -s /test
Authentication is not valid : /test
[zk: 127.0.0.1:2181(CONNECTED) 1] getAcl /test
'ip,'127.0.0.1
: cdrwa
[zk: 127.0.0.1:2181(CONNECTED) 2] get /test
Authentication is not valid : /test
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 26 weeks, 1 day ago 0|i33qsf:
ZooKeeper ZOOKEEPER-2587

Not handled negative scenario for redo command

Bug Resolved Minor Duplicate Rakesh Kumar Singh Rakesh Kumar Singh Rakesh Kumar Singh 16/Sep/16 01:44   06/Oct/16 07:31 06/Oct/16 07:31     java client   0 2   [zk: localhost:2181(CONNECTED) 2] redo -1
Exception in thread "main" java.lang.NullPointerException
at java.util.StringTokenizer.<init>(StringTokenizer.java:199)
at java.util.StringTokenizer.<init>(StringTokenizer.java:221)
at org.apache.zookeeper.ZooKeeperMain$MyCommandOptions.parseCommand(ZooKeeperMain.java:219)
at org.apache.zookeeper.ZooKeeperMain.processZKCmd(ZooKeeperMain.java:638)
at org.apache.zookeeper.ZooKeeperMain.processCmd(ZooKeeperMain.java:577)
at org.apache.zookeeper.ZooKeeperMain.executeLine(ZooKeeperMain.java:360)
at org.apache.zookeeper.ZooKeeperMain.run(ZooKeeperMain.java:320)
at org.apache.zookeeper.ZooKeeperMain.main(ZooKeeperMain.java:280)
9223372036854775807 No Perforce job exists for this issue. 2 9223372036854775807
3 years, 24 weeks ago 0|i33p4n:
ZooKeeper ZOOKEEPER-2586

zoo_aremove_watchers() does not remove a watch of path has more than one watch

Bug Open Major Unresolved Unassigned prashant prashant 16/Sep/16 00:35   16/Dec/18 09:46           1 3   three issues
1)zoo_aremove_watchers() does not remove a watch if path has more than one watch.
but it works in below cases.
it removes watch if path has only one watch.
and it removes all watches if watcher function arguments is NULL.
Seen in
zookeeper.version=3.5.1-alpha--1, built on 06/09/2016 18:31 GMT
Not sure if this is fixed in later versions.

2) If zoo_aremove_watchers() is called with local=1, then client hangs in waiting for mutex in mt_adaptor.c:102

void notify_sync_completion(struct sync_completion *sc)
{
pthread_mutex_lock(&sc->lock);
...

3) Acts like sync API if no node and no watcher on path.
it does not call async completion callback in this case.
remove_watches 9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
2 years, 51 weeks, 4 days ago 0|i33p33:
ZooKeeper ZOOKEEPER-2585

ACL with SSL is not working

Bug Open Critical Unresolved Unassigned Rakesh Kumar Singh Rakesh Kumar Singh 15/Sep/16 03:16   21/Sep/16 04:48   3.5.1   server   0 3   Set ACL with SSL is not working

Steps to reproduce:-
1. Start zookeeper in ssl mode in standalone
2. Connect zookeeper from zookeeper client (using zkCli.sh)
3. add auth and set ACL as below and then quit the client :-

[zk: localhost:2181(CONNECTED) 0] addauth digest u1:p1
[zk: localhost:2181(CONNECTED) 1] create /test_auth hello
Created /test_auth
[zk: localhost:2181(CONNECTED) 2] setAcl /test_auth auth:u1:p1:crdwa
[zk: localhost:2181(CONNECTED) 3] get /test_auth
hello
[zk: localhost:2181(CONNECTED) 4] quit

4. Connect again zookeeper from zookeeper client (using zkCli.sh)
5. Try to access the znode, try to set the data and so on, everything is allowed

[zk: localhost:2181(CONNECTED) 2] set /test_auth hello1
[zk: localhost:2181(CONNECTED) 3] get /test_auth
hello1
[zk: localhost:2181(CONNECTED) 1] getAcl /test_auth
'x509,'CN=locahost%2COU=CS%2CO=HUAWEI%2CL=Shenzhen%2CST=Guangdong%2CC=CHINA
: cdrwa
'digest,'u1:fpT/y03U+EjItKZOSLGvjnJlyng=
: cdrwa
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 26 weeks, 1 day ago 0|i33nl3:
ZooKeeper ZOOKEEPER-2584

when setquota for a znode and set ip/user ACL on /zookeeper/quota, still able to delete the quota from client with another ip though it says "Authentication is not valid"

Bug Open Major Unresolved Rakesh Kumar Singh Rakesh Kumar Singh Rakesh Kumar Singh 15/Sep/16 00:48   28/May/19 07:03   3.5.1   server   0 2   when setquota for a znode and set ip/user ACL on /zookeeper/quota, still able to delete the quota from client with another ip though it says "Authentication is not valid"


>> Set quota and ip ACL from one client (with IP 10.18.101.80)
[zk: 10.18.101.80:2181(CONNECTED) 9] setquota -n 10 /test
[zk: 10.18.101.80:2181(CONNECTED) 10] setAcl /zookeeper/quota ip:10.18.101.80:crdwa
[zk: 10.18.101.80:2181(CONNECTED) 11]


>> Try to delete the set quota using different client(with ip 10.18.219.50)
[zk: 10.18.219.50:2181(CONNECTED) 22] listquota /test
absolute path is /zookeeper/quota/test/zookeeper_limits
Output quota for /test count=10,bytes=-1
Output stat for /test count=1,bytes=5
[zk: 10.18.219.50:2181(CONNECTED) 23] delquota /test
Authentication is not valid : /zookeeper/quota/test
[zk: 10.18.219.50:2181(CONNECTED) 24] listquota /test
absolute path is /zookeeper/quota/test/zookeeper_limits
quota for /test does not exist.

>> Here quota has been deleted though it is saying "Authentication is not valid.." which is not correct.
Now try to set the quota from another ip itself, it fails which is as expected

[zk: 10.18.219.50:2181(CONNECTED) 25] setquota -n 10 /test
Authentication is not valid : /zookeeper/quota/test
[zk: 10.18.219.50:2181(CONNECTED) 26] listquota /test
absolute path is /zookeeper/quota/test/zookeeper_limits
quota for /test does not exist.


>> Sameway when we set user ACL...
[zk: 10.18.101.80:2181(CONNECTED) 26] addauth digest user:pass
[zk: 10.18.101.80:2181(CONNECTED) 27] create /test hello
Node already exists: /test
[zk: 10.18.101.80:2181(CONNECTED) 28] delete /test
[zk: 10.18.101.80:2181(CONNECTED) 29] create /test hello
Created /test
[zk: 10.18.101.80:2181(CONNECTED) 30]
[zk: 10.18.101.80:2181(CONNECTED) 30] setquota -n 10 /test
[zk: 10.18.101.80:2181(CONNECTED) 31] setAcl /zookeeper/quota auth:user:pass:crdwa
[zk: 10.18.101.80:2181(CONNECTED) 32]


[zk: 10.18.219.50:2181(CONNECTED) 27] listquota /test
absolute path is /zookeeper/quota/test/zookeeper_limits
Output quota for /test count=10,bytes=-1
Output stat for /test count=1,bytes=5
[zk: 10.18.219.50:2181(CONNECTED) 28] delquota /test
Authentication is not valid : /zookeeper/quota/test
[zk: 10.18.219.50:2181(CONNECTED) 29] listquota /test
absolute path is /zookeeper/quota/test/zookeeper_limits
quota for /test does not exist.
[zk: 10.18.219.50:2181(CONNECTED) 30]
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
42 weeks, 2 days ago 0|i33nhb:
ZooKeeper ZOOKEEPER-2583

Using one client able to access the znode with localhost but fails from another client when IP ACL is set for znode using 127.0.0.1

Bug Open Minor Unresolved Rakesh Kumar Singh Rakesh Kumar Singh Rakesh Kumar Singh 14/Sep/16 10:30   23/Sep/16 01:04   3.5.1   server   0 3   Using one client able to access the znode with localhost but fails from another client when IP ACL is set for znode using 127.0.0.1


Start zookeeper in cluster mode.

Client 1 :-

[zk: localhost:2181(CONNECTED) 11] create /ip_test hello
Created /ip_test
[zk: localhost:2181(CONNECTED) 12] setAcl /ip_test
ip_test ip_test4
[zk: localhost:2181(CONNECTED) 12] setAcl /ip_test ip:127.0.0.1:crdwa
[zk: localhost:2181(CONNECTED) 13] get /ip_test
hello
[zk: localhost:2181(CONNECTED) 14] set /ip_test hi
[zk: localhost:2181(CONNECTED) 15]


Client 2 :-

[zk: localhost:2181(CONNECTED) 0] get /ip_test
Authentication is not valid : /ip_test
[zk: localhost:2181(CONNECTED) 1] getAcl /ip_test
'ip,'127.0.0.1
: cdrwa

[zk: localhost:2181(CONNECTED) 2] quit

now quit the client connection and connect again using 127.0.0.1 (like :- ./zkCli.sh -server 127.0.0.1:2181)

[zk: 127.0.0.1:2181(CONNECTED) 0] get /ip_test
hi
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 27 weeks, 1 day ago 0|i33mg7:
ZooKeeper ZOOKEEPER-2582

When addauth twice for same user but different password, it is adding 2 digest corresponding to both username, password and so we can able to access znode with user and any of these password which does not seem to be correct

Bug Open Major Unresolved Unassigned Rakesh Kumar Singh Rakesh Kumar Singh 14/Sep/16 09:38   20/Sep/16 15:29   3.5.1   server   0 3   When addauth twice for same user but different password, it is adding 2 digest corresponding to both username, password and so we can able to access znode with user and any of these password which does not seem to be correct

Steps:-
[zk: localhost:2181(CONNECTED) 0] addauth digest user1:pass1
[zk: localhost:2181(CONNECTED) 1] addauth digest user1:pass

[zk: localhost:2181(CONNECTED) 9] create /user_test5 hello
Created /user_test5
[zk: localhost:2181(CONNECTED) 10] setAcl /user_test5 auth:user1:pass1:crdwa
[zk: localhost:2181(CONNECTED) 11] getAcl /user_test5
'digest,'user1:+7K83PhyQ3ijGj0ADmljf0quVwQ=
: cdrwa
'digest,'user1:UZIsvOKp29j8vAahJzjgpA1VTOk=
: cdrwa


Here we can see 2 entries for same user (user1) with different password

Now disconnect the client and connect again using zkCli.sh
addauth digest user1:<any of 2 password>, we can able to access the znode.

[zk: localhost:2181(CONNECTED) 0] get /user_test5
Authentication is not valid : /user_test5
[zk: localhost:2181(CONNECTED) 1] addauth digest user1:pass
[zk: localhost:2181(CONNECTED) 2] get /user_test5
hello

Same way, it will allow n number of entry if we addauth for same user with n number of password
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 26 weeks, 2 days ago 0|i33mc7:
ZooKeeper ZOOKEEPER-2581

Not handled NullPointerException while creating key manager and trustManager

Bug Resolved Major Fixed maoling Rakesh Kumar Singh Rakesh Kumar Singh 14/Sep/16 03:12   11/Sep/17 17:37 11/Sep/17 16:57 3.5.1 3.5.4, 3.6.0 server   1 6   Not handled NullPointerException while creating key manager and trustManager:-


2016-09-14 13:35:23,488 [myid:1] - ERROR [CommitProcWorkThread-1:X509AuthenticationProvider@78] - Failed to create key manager
org.apache.zookeeper.common.X509Exception$KeyManagerException: java.lang.NullPointerException
at org.apache.zookeeper.common.X509Util.createKeyManager(X509Util.java:129)
at org.apache.zookeeper.server.auth.X509AuthenticationProvider.<init>(X509AuthenticationProvider.java:75)
at org.apache.zookeeper.server.auth.ProviderRegistry.initialize(ProviderRegistry.java:42)
at org.apache.zookeeper.server.auth.ProviderRegistry.getProvider(ProviderRegistry.java:68)
at org.apache.zookeeper.server.PrepRequestProcessor.checkACL(PrepRequestProcessor.java:319)
at org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:324)
at org.apache.zookeeper.server.quorum.CommitProcessor$CommitWorkRequest.doWork(CommitProcessor.java:296)
at org.apache.zookeeper.server.WorkerService$ScheduledWorkRequest.run(WorkerService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NullPointerException
at org.apache.zookeeper.common.X509Util.createKeyManager(X509Util.java:113)
... 10 more
2016-09-14 13:35:23,489 [myid:1] - ERROR [CommitProcWorkThread-1:X509AuthenticationProvider@90] - Failed to create trust manager
org.apache.zookeeper.common.X509Exception$TrustManagerException: java.lang.NullPointerException
at org.apache.zookeeper.common.X509Util.createTrustManager(X509Util.java:158)
at org.apache.zookeeper.server.auth.X509AuthenticationProvider.<init>(X509AuthenticationProvider.java:87)
at org.apache.zookeeper.server.auth.ProviderRegistry.initialize(ProviderRegistry.java:42)
at org.apache.zookeeper.server.auth.ProviderRegistry.getProvider(ProviderRegistry.java:68)
at org.apache.zookeeper.server.PrepRequestProcessor.checkACL(PrepRequestProcessor.java:319)
at org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:324)
at org.apache.zookeeper.server.quorum.CommitProcessor$CommitWorkRequest.doWork(CommitProcessor.java:296)
at org.apache.zookeeper.server.WorkerService$ScheduledWorkRequest.run(WorkerService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NullPointerException
at org.apache.zookeeper.common.X509Util.createTrustManager(X509Util.java:143)
... 10 more
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
2 years, 27 weeks, 3 days ago 0|i33lrb:
ZooKeeper ZOOKEEPER-2580

ErrorMessage is not correct when set IP acl and try to set again from another machine

Bug Open Minor Unresolved Rakesh Kumar Singh Rakesh Kumar Singh Rakesh Kumar Singh 14/Sep/16 00:48   23/Sep/16 01:04   3.5.1   java client   0 3   set IP acl and try to set again from another machine:-

[zk: localhost:2181(CONNECTED) 11] setAcl /ip_test ip:10.18.101.80:crdwa
KeeperErrorCode = NoAuth for /ip_test
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 26 weeks ago 0|i33lnj:
ZooKeeper ZOOKEEPER-2579

ZooKeeper server should verify that dataDir and snapDir are writeable before starting

Bug Closed Major Fixed Abraham Fine Abraham Fine Abraham Fine 13/Sep/16 16:49   31/Mar/17 05:01 18/Sep/16 17:37 3.4.9, 3.5.2 3.4.10, 3.5.3, 3.6.0     0 4   If the directories specified for the dataDir or the snapDir are not writeable, the server does not fail until it actually tries to write there. It should fail when it starts. 9223372036854775807 No Perforce job exists for this issue. 5 9223372036854775807
3 years, 26 weeks, 4 days ago
Reviewed
0|i33l9j:
ZooKeeper ZOOKEEPER-2578

zkEnv.sh does not set $ZOOCFG properly if already set

Bug Open Minor Unresolved Unassigned Bjorn Stange Bjorn Stange 13/Sep/16 16:30   21/Sep/16 11:37   3.2.0, 3.2.1, 3.2.2, 3.2.3, 3.3.0, 3.3.1, 3.3.2, 3.3.3, 3.3.4, 3.4.0, 3.4.1, 3.4.2, 3.4.3, 3.3.5, 3.3.6, 3.4.4, 3.4.5, 3.4.6, 3.4.7, 3.4.8, 3.4.9, 3.4.10, 3.5.0, 3.5.1, 3.5.2       0 3   In bin/zkEnv.sh The ZOOCFG variable is duplicated over itself if already set. For example, in my use case it was being set in zookeeper-env.sh. The problem seems to arise from this line (line 61 on the master branch at the time of this submission): ZOOCFG="$ZOOCFGDIR/$ZOOCFG". This overwrites the value of ZOOCFG by appending the old value to the value of ZOOCFGDIR, which is problematic if it was already initialized as the absolute path to a file. The behavior of overwriting the value of the variable in this way seems to be specific to the case where ZOOCFG is not initialized. The final state of ZOOCFG seemingly is the absolute path to the zookeeper configuration file. This behavior assumes that it is the filename only, which is where the bug arises. 9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
Patch
3 years, 26 weeks, 1 day ago zkEnv no longer erroneously overwrites ZOOCFG 0|i33l8v:
ZooKeeper ZOOKEEPER-2577

Flaky Test: org.apache.zookeeper.server.quorum.ReconfigDuringLeaderSyncTest.testDuringLeaderSync

Test Resolved Major Fixed Michael Han Michael Han Michael Han 13/Sep/16 14:33   27/Jul/17 17:45 27/Jul/17 17:14 3.5.2 3.5.4, 3.6.0 tests   0 4  
{noformat}
Error Message

zoo.cfg.dynamic.next is not deleted.
Stacktrace

junit.framework.AssertionFailedError: zoo.cfg.dynamic.next is not deleted.
at org.apache.zookeeper.server.quorum.ReconfigDuringLeaderSyncTest.testDuringLeaderSync(ReconfigDuringLeaderSyncTest.java:155)
at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:79)
Standard Output

2016-09-13 05:09:25,247 [myid:] - INFO [main:JUnit4ZKTestRunner@47] - No test.method specified. using default methods.
2016-09-13 05:09:25,349 [myid:] - INFO [main:JUnit4ZKTestRunner@47] - No test.method specified. using default methods.
2016-09-13 05:09:25,370 [myid:] - INFO [main:ZKTestCase$1@55] - STARTING testDuringLeaderSync
2016-09-13 05:09:25,372 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@77] - RUNNING TEST METHOD testDuringLeaderSync
2016-09-13 05:09:25,375 [myid:] - INFO [main:PortAssignment@151] - Test process 2/8 using ports from 13914 - 16606.
2016-09-13 05:09:25,380 [myid:] - INFO [main:PortAssignment@85] - Assigned port 13915 from range 13914 - 16606.
2016-09-13 05:09:25,380 [myid:] - INFO [main:PortAssignment@85] - Assigned port 13916 from range 13914 - 16606.
2016-09-13 05:09:25,381 [myid:] - INFO [main:PortAssignment@85] - Assigned port 13917 from range 13914 - 16606.
2016-09-13 05:09:25,381 [myid:] - INFO [main:PortAssignment@85] - Assigned port 13918 from range 13914 - 16606.
2016-09-13 05:09:25,381 [myid:] - INFO [main:PortAssignment@85] - Assigned port 13919 from range 13914 - 16606.
2016-09-13 05:09:25,382 [myid:] - INFO [main:PortAssignment@85] - Assigned port 13920 from range 13914 - 16606.
2016-09-13 05:09:25,382 [myid:] - INFO [main:PortAssignment@85] - Assigned port 13921 from range 13914 - 16606.
2016-09-13 05:09:25,382 [myid:] - INFO [main:PortAssignment@85] - Assigned port 13922 from range 13914 - 16606.
2016-09-13 05:09:25,383 [myid:] - INFO [main:PortAssignment@85] - Assigned port 13923 from range 13914 - 16606.
2016-09-13 05:09:25,406 [myid:] - INFO [main:QuorumPeerTestBase$MainThread@131] - id = 0 tmpDir = /home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/test/tmp/test8397079557861207505.junit.dir clientPort = 13915 adminServerPort = 8080
2016-09-13 05:09:25,416 [myid:] - INFO [main:QuorumPeerTestBase$MainThread@131] - id = 1 tmpDir = /home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/test/tmp/test1768888919429940621.junit.dir clientPort = 13918 adminServerPort = 8080
2016-09-13 05:09:25,420 [myid:] - INFO [main:QuorumPeerTestBase$MainThread@131] - id = 2 tmpDir = /home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/test/tmp/test5455612786130415623.junit.dir clientPort = 13921 adminServerPort = 8080
2016-09-13 05:09:25,422 [myid:] - INFO [Thread-0:QuorumPeerConfig@116] - Reading configuration from: /home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/test/tmp/test8397079557861207505.junit.dir/zoo.cfg
2016-09-13 05:09:25,422 [myid:] - INFO [Thread-2:QuorumPeerConfig@116] - Reading configuration from: /home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/test/tmp/test5455612786130415623.junit.dir/zoo.cfg
2016-09-13 05:09:25,422 [myid:] - INFO [Thread-1:QuorumPeerConfig@116] - Reading configuration from: /home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/test/tmp/test1768888919429940621.junit.dir/zoo.cfg
2016-09-13 05:09:25,424 [myid:] - INFO [main:FourLetterWordMain@85] - connecting to 127.0.0.1 13915
2016-09-13 05:09:25,425 [myid:] - INFO [Thread-0:QuorumPeerConfig@318] - clientPortAddress is 0.0.0.0/0.0.0.0:13915
2016-09-13 05:09:25,425 [myid:] - INFO [Thread-0:QuorumPeerConfig@322] - secureClientPort is not set
2016-09-13 05:09:25,425 [myid:] - INFO [Thread-1:QuorumPeerConfig@318] - clientPortAddress is 0.0.0.0/0.0.0.0:13918
2016-09-13 05:09:25,425 [myid:] - INFO [Thread-1:QuorumPeerConfig@322] - secureClientPort is not set
2016-09-13 05:09:25,425 [myid:] - INFO [Thread-2:QuorumPeerConfig@318] - clientPortAddress is 0.0.0.0/0.0.0.0:13921
2016-09-13 05:09:25,426 [myid:] - INFO [Thread-2:QuorumPeerConfig@322] - secureClientPort is not set
2016-09-13 05:09:25,430 [myid:] - INFO [main:ClientBase@248] - server 127.0.0.1:13915 not up
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.client.FourLetterWordMain.send4LetterWord(FourLetterWordMain.java:99)
at org.apache.zookeeper.client.FourLetterWordMain.send4LetterWord(FourLetterWordMain.java:69)
at org.apache.zookeeper.test.ClientBase.waitForServerUp(ClientBase.java:241)
at org.apache.zookeeper.test.ClientBase.waitForServerUp(ClientBase.java:232)
at org.apache.zookeeper.server.quorum.ReconfigDuringLeaderSyncTest.testDuringLeaderSync(ReconfigDuringLeaderSyncTest.java:85)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:79)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.TestWatchman$1.evaluate(TestWatchman.java:53)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:38)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:535)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1182)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:1033)
2016-09-13 05:09:25,444 [myid:1] - INFO [Thread-1:DatadirCleanupManager@78] - autopurge.snapRetainCount set to 3
2016-09-13 05:09:25,445 [myid:2] - INFO [Thread-2:DatadirCleanupManager@78] - autopurge.snapRetainCount set to 3
2016-09-13 05:09:25,445 [myid:2] - INFO [Thread-2:DatadirCleanupManager@79] - autopurge.purgeInterval set to 0
2016-09-13 05:09:25,444 [myid:0] - INFO [Thread-0:DatadirCleanupManager@78] - autopurge.snapRetainCount set to 3
2016-09-13 05:09:25,445 [myid:2] - INFO [Thread-2:DatadirCleanupManager@101] - Purge task is not scheduled.
2016-09-13 05:09:25,445 [myid:1] - INFO [Thread-1:DatadirCleanupManager@79] - autopurge.purgeInterval set to 0
2016-09-13 05:09:25,446 [myid:1] - INFO [Thread-1:DatadirCleanupManager@101] - Purge task is not scheduled.
2016-09-13 05:09:25,445 [myid:0] - INFO [Thread-0:DatadirCleanupManager@79] - autopurge.purgeInterval set to 0
2016-09-13 05:09:25,446 [myid:0] - INFO [Thread-0:DatadirCleanupManager@101] - Purge task is not scheduled.
2016-09-13 05:09:25,446 [myid:2] - INFO [Thread-2:ManagedUtil@46] - Log4j found with jmx enabled.
2016-09-13 05:09:25,447 [myid:0] - INFO [Thread-0:ManagedUtil@46] - Log4j found with jmx enabled.
2016-09-13 05:09:25,446 [myid:1] - INFO [Thread-1:ManagedUtil@46] - Log4j found with jmx enabled.
2016-09-13 05:09:25,552 [myid:1] - ERROR [Thread-1:ManagedUtil@114] - Problems while registering log4j jmx beans!
javax.management.InstanceAlreadyExistsException: log4j:hiearchy=default
at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:437)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1898)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:966)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324)
at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
at org.apache.zookeeper.jmx.ManagedUtil.registerLog4jMBeans(ManagedUtil.java:75)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:131)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:120)
at org.apache.zookeeper.server.quorum.QuorumPeerTestBase$MainThread.run(QuorumPeerTestBase.java:245)
at java.lang.Thread.run(Thread.java:745)
2016-09-13 05:09:25,552 [myid:1] - WARN [Thread-1:QuorumPeerMain@133] - Unable to register log4j JMX control
javax.management.JMException: javax.management.InstanceAlreadyExistsException: log4j:hiearchy=default
at org.apache.zookeeper.jmx.ManagedUtil.registerLog4jMBeans(ManagedUtil.java:115)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:131)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:120)
at org.apache.zookeeper.server.quorum.QuorumPeerTestBase$MainThread.run(QuorumPeerTestBase.java:245)
at java.lang.Thread.run(Thread.java:745)
2016-09-13 05:09:25,553 [myid:1] - INFO [Thread-1:QuorumPeerMain@136] - Starting quorum peer
2016-09-13 05:09:25,553 [myid:2] - ERROR [Thread-2:HierarchyDynamicMBean@138] - Could not add loggerMBean for [root].
javax.management.InstanceAlreadyExistsException: log4j:logger=root
at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:437)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1898)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:966)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324)
at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
at org.apache.log4j.jmx.AbstractDynamicMBean.registerMBean(AbstractDynamicMBean.java:160)
at org.apache.log4j.jmx.HierarchyDynamicMBean.addLoggerMBean(HierarchyDynamicMBean.java:125)
at org.apache.log4j.jmx.HierarchyDynamicMBean.postRegister(HierarchyDynamicMBean.java:263)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.postRegister(DefaultMBeanServerInterceptor.java:1024)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:974)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324)
at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
at org.apache.zookeeper.jmx.ManagedUtil.registerLog4jMBeans(ManagedUtil.java:75)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:131)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:120)
at org.apache.zookeeper.server.quorum.QuorumPeerTestBase$MainThread.run(QuorumPeerTestBase.java:245)
at java.lang.Thread.run(Thread.java:745)
2016-09-13 05:09:25,559 [myid:0] - ERROR [Thread-0:ManagedUtil@114] - Problems while registering log4j jmx beans!
javax.management.InstanceAlreadyExistsException: log4j:hiearchy=default
at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:437)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1898)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:966)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324)
at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
at org.apache.zookeeper.jmx.ManagedUtil.registerLog4jMBeans(ManagedUtil.java:75)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:131)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:120)
at org.apache.zookeeper.server.quorum.QuorumPeerTestBase$MainThread.run(QuorumPeerTestBase.java:245)
at java.lang.Thread.run(Thread.java:745)
2016-09-13 05:09:25,560 [myid:0] - WARN [Thread-0:QuorumPeerMain@133] - Unable to register log4j JMX control
javax.management.JMException: javax.management.InstanceAlreadyExistsException: log4j:hiearchy=default
at org.apache.zookeeper.jmx.ManagedUtil.registerLog4jMBeans(ManagedUtil.java:115)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:131)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:120)
at org.apache.zookeeper.server.quorum.QuorumPeerTestBase$MainThread.run(QuorumPeerTestBase.java:245)
at java.lang.Thread.run(Thread.java:745)
2016-09-13 05:09:25,560 [myid:0] - INFO [Thread-0:QuorumPeerMain@136] - Starting quorum peer
2016-09-13 05:09:25,562 [myid:2] - INFO [Thread-2:QuorumPeerMain@136] - Starting quorum peer
2016-09-13 05:09:25,581 [myid:0] - INFO [Thread-0:NIOServerCnxnFactory@673] - Configuring NIO connection handler with 10s sessionless connection timeout, 1 selector thread(s), 8 worker threads, and 64 kB direct buffers.
2016-09-13 05:09:25,581 [myid:2] - INFO [Thread-2:NIOServerCnxnFactory@673] - Configuring NIO connection handler with 10s sessionless connection timeout, 1 selector thread(s), 8 worker threads, and 64 kB direct buffers.
2016-09-13 05:09:25,585 [myid:1] - INFO [Thread-1:NIOServerCnxnFactory@673] - Configuring NIO connection handler with 10s sessionless connection timeout, 1 selector thread(s), 8 worker threads, and 64 kB direct buffers.
2016-09-13 05:09:25,591 [myid:0] - INFO [Thread-0:NIOServerCnxnFactory@686] - binding to port /127.0.0.1:13915
2016-09-13 05:09:25,592 [myid:1] - INFO [Thread-1:NIOServerCnxnFactory@686] - binding to port /127.0.0.1:13918
2016-09-13 05:09:25,593 [myid:2] - INFO [Thread-2:NIOServerCnxnFactory@686] - binding to port /127.0.0.1:13921
2016-09-13 05:09:25,622 [myid:2] - INFO [Thread-2:QuorumPeer@1327] - Local sessions disabled
2016-09-13 05:09:25,622 [myid:2] - INFO [Thread-2:QuorumPeer@1338] - Local session upgrading disabled
2016-09-13 05:09:25,622 [myid:2] - INFO [Thread-2:QuorumPeer@1305] - tickTime set to 4000
2016-09-13 05:09:25,622 [myid:2] - INFO [Thread-2:QuorumPeer@1349] - minSessionTimeout set to 8000
2016-09-13 05:09:25,622 [myid:2] - INFO [Thread-2:QuorumPeer@1360] - maxSessionTimeout set to 80000
2016-09-13 05:09:25,622 [myid:1] - INFO [Thread-1:QuorumPeer@1327] - Local sessions disabled
2016-09-13 05:09:25,622 [myid:1] - INFO [Thread-1:QuorumPeer@1338] - Local session upgrading disabled
2016-09-13 05:09:25,623 [myid:1] - INFO [Thread-1:QuorumPeer@1305] - tickTime set to 4000
2016-09-13 05:09:25,623 [myid:1] - INFO [Thread-1:QuorumPeer@1349] - minSessionTimeout set to 8000
2016-09-13 05:09:25,623 [myid:1] - INFO [Thread-1:QuorumPeer@1360] - maxSessionTimeout set to 80000
2016-09-13 05:09:25,623 [myid:1] - INFO [Thread-1:QuorumPeer@1375] - initLimit set to 10
2016-09-13 05:09:25,622 [myid:0] - INFO [Thread-0:QuorumPeer@1327] - Local sessions disabled
2016-09-13 05:09:25,625 [myid:0] - INFO [Thread-0:QuorumPeer@1338] - Local session upgrading disabled
2016-09-13 05:09:25,625 [myid:0] - INFO [Thread-0:QuorumPeer@1305] - tickTime set to 4000
2016-09-13 05:09:25,625 [myid:0] - INFO [Thread-0:QuorumPeer@1349] - minSessionTimeout set to 8000
2016-09-13 05:09:25,625 [myid:0] - INFO [Thread-0:QuorumPeer@1360] - maxSessionTimeout set to 80000
2016-09-13 05:09:25,625 [myid:0] - INFO [Thread-0:QuorumPeer@1375] - initLimit set to 10
2016-09-13 05:09:25,622 [myid:2] - INFO [Thread-2:QuorumPeer@1375] - initLimit set to 10
2016-09-13 05:09:25,666 [myid:0] - INFO [Thread-0:QuorumPeer@776] - currentEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
2016-09-13 05:09:25,668 [myid:1] - INFO [Thread-1:QuorumPeer@776] - currentEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
2016-09-13 05:09:25,668 [myid:2] - INFO [Thread-2:QuorumPeer@776] - currentEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
2016-09-13 05:09:25,669 [myid:0] - INFO [Thread-0:QuorumPeer@791] - acceptedEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
2016-09-13 05:09:25,672 [myid:1] - INFO [Thread-1:QuorumPeer@791] - acceptedEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
2016-09-13 05:09:25,677 [myid:2] - INFO [Thread-2:QuorumPeer@791] - acceptedEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
2016-09-13 05:09:25,689 [myid:] - INFO [main:FourLetterWordMain@85] - connecting to 127.0.0.1 13915
2016-09-13 05:09:25,704 [myid:0] - INFO [NIOServerCxnFactory.AcceptThread:/127.0.0.1:13915:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:38366
2016-09-13 05:09:25,758 [myid:0] - INFO [NIOWorkerThread-1:NIOServerCnxn@485] - Processing stat command from /127.0.0.1:38366
2016-09-13 05:09:25,770 [myid:1] - INFO [QuorumPeerListener:QuorumCnxManager$Listener@632] - My election bind port: /127.0.0.1:13920
2016-09-13 05:09:25,772 [myid:0] - INFO [NIOWorkerThread-1:NIOServerCnxn@607] - Closed socket connection for client /127.0.0.1:38366 (no session established for client)
2016-09-13 05:09:25,783 [myid:1] - INFO [QuorumPeer[myid=1](plain=/127.0.0.1:13918)(secure=disabled):QuorumPeer@1033] - LOOKING
2016-09-13 05:09:25,785 [myid:1] - INFO [QuorumPeer[myid=1](plain=/127.0.0.1:13918)(secure=disabled):FastLeaderElection@894] - New election. My id = 1, proposed zxid=0x0
2016-09-13 05:09:25,797 [myid:0] - INFO [QuorumPeerListener:QuorumCnxManager$Listener@632] - My election bind port: /127.0.0.1:13917
2016-09-13 05:09:25,798 [myid:0] - INFO [QuorumPeer[myid=0](plain=/127.0.0.1:13915)(secure=disabled):QuorumPeer@1033] - LOOKING
2016-09-13 05:09:25,798 [myid:0] - INFO [QuorumPeer[myid=0](plain=/127.0.0.1:13915)(secure=disabled):FastLeaderElection@894] - New election. My id = 0, proposed zxid=0x0
2016-09-13 05:09:25,799 [myid:0] - INFO [/127.0.0.1:13917:QuorumCnxManager$Listener@638] - Received connection request /127.0.0.1:53023
2016-09-13 05:09:25,806 [myid:2] - INFO [QuorumPeer[myid=2](plain=/127.0.0.1:13921)(secure=disabled):QuorumPeer@1033] - LOOKING
2016-09-13 05:09:25,806 [myid:2] - INFO [QuorumPeerListener:QuorumCnxManager$Listener@632] - My election bind port: /127.0.0.1:13923
2016-09-13 05:09:25,806 [myid:2] - INFO [QuorumPeer[myid=2](plain=/127.0.0.1:13921)(secure=disabled):FastLeaderElection@894] - New election. My id = 2, proposed zxid=0x0
2016-09-13 05:09:25,825 [myid:2] - INFO [/127.0.0.1:13923:QuorumCnxManager$Listener@638] - Received connection request /127.0.0.1:58679
2016-09-13 05:09:25,826 [myid:0] - INFO [WorkerSender[myid=0]:QuorumCnxManager@276] - Have smaller server identifier, so dropping the connection: (2, 0)
2016-09-13 05:09:25,825 [myid:1] - INFO [WorkerSender[myid=1]:QuorumCnxManager@276] - Have smaller server identifier, so dropping the connection: (2, 1)
2016-09-13 05:09:25,827 [myid:0] - INFO [/127.0.0.1:13917:QuorumCnxManager$Listener@638] - Received connection request /127.0.0.1:53026
2016-09-13 05:09:25,827 [myid:0] - INFO [WorkerReceiver[myid=0]:FastLeaderElection@688] - Notification: 2 (message format version), 0 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 0 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version)
2016-09-13 05:09:25,827 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@688] - Notification: 2 (message format version), 1 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version)
2016-09-13 05:09:25,848 [myid:1] - INFO [/127.0.0.1:13920:QuorumCnxManager$Listener@638] - Received connection request /127.0.0.1:46423
2016-09-13 05:09:25,848 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@688] - Notification: 2 (message format version), 0 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 0 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version)
2016-09-13 05:09:25,857 [myid:2] - INFO [/127.0.0.1:13923:QuorumCnxManager$Listener@638] - Received connection request /127.0.0.1:58680
2016-09-13 05:09:25,857 [myid:0] - INFO [WorkerReceiver[myid=0]:FastLeaderElection@688] - Notification: 2 (message format version), 1 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version)
2016-09-13 05:09:25,858 [myid:2] - WARN [RecvWorker:0:QuorumCnxManager$RecvWorker@915] - Connection broken for id 0, my id = 2, error =
java.net.SocketException: Socket closed
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at java.net.SocketInputStream.read(SocketInputStream.java:210)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:900)
2016-09-13 05:09:25,858 [myid:2] - WARN [RecvWorker:0:QuorumCnxManager$RecvWorker@918] - Interrupting SendWorker
2016-09-13 05:09:25,862 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@688] - Notification: 2 (message format version), 1 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 0 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version)
2016-09-13 05:09:25,863 [myid:0] - INFO [WorkerReceiver[myid=0]:FastLeaderElection@688] - Notification: 2 (message format version), 1 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 0 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version)
2016-09-13 05:09:25,865 [myid:0] - WARN [SendWorker:2:QuorumCnxManager$SendWorker@837] - Exception when using channel: for id 2 my id = 0 error = java.net.SocketException: Broken pipe
2016-09-13 05:09:25,867 [myid:0] - WARN [RecvWorker:2:QuorumCnxManager$RecvWorker@918] - Interrupting SendWorker
2016-09-13 05:09:25,898 [myid:2] - WARN [SendWorker:0:QuorumCnxManager$SendWorker@832] - Interrupted while waiting for message on queue
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:982)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:63)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:820)
2016-09-13 05:09:25,898 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@688] - Notification: 2 (message format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version)
2016-09-13 05:09:25,898 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@688] - Notification: 2 (message format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version)
2016-09-13 05:09:25,899 [myid:0] - WARN [SendWorker:2:QuorumCnxManager$SendWorker@841] - Send worker leaving thread id 2 my id = 0
2016-09-13 05:09:25,899 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@688] - Notification: 2 (message format version), 1 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version)
2016-09-13 05:09:25,900 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@688] - Notification: 2 (message format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version)
2016-09-13 05:09:25,901 [myid:0] - INFO [WorkerReceiver[myid=0]:FastLeaderElection@688] - Notification: 2 (message format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version)
2016-09-13 05:09:25,901 [myid:0] - INFO [/127.0.0.1:13917:QuorumCnxManager$Listener@638] - Received connection request /127.0.0.1:53028
2016-09-13 05:09:25,898 [myid:2] - WARN [SendWorker:0:QuorumCnxManager$SendWorker@841] - Send worker leaving thread id 0 my id = 2
2016-09-13 05:09:25,901 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@688] - Notification: 2 (message format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version)
2016-09-13 05:09:25,902 [myid:0] - INFO [WorkerSender[myid=0]:QuorumCnxManager@276] - Have smaller server identifier, so dropping the connection: (2, 0)
2016-09-13 05:09:25,903 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@688] - Notification: 2 (message format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 0 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version)
2016-09-13 05:09:25,903 [myid:0] - INFO [WorkerReceiver[myid=0]:FastLeaderElection@688] - Notification: 2 (message format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 0 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version)
2016-09-13 05:09:25,904 [myid:2] - INFO [/127.0.0.1:13923:QuorumCnxManager$Listener@638] - Received connection request /127.0.0.1:58684
2016-09-13 05:09:25,905 [myid:2] - WARN [SendWorker:0:QuorumCnxManager$SendWorker@832] - Interrupted while waiting for message on queue
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:982)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:63)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:820)
2016-09-13 05:09:25,905 [myid:2] - WARN [SendWorker:0:QuorumCnxManager$SendWorker@841] - Send worker leaving thread id 0 my id = 2
2016-09-13 05:09:25,905 [myid:2] - WARN [RecvWorker:0:QuorumCnxManager$RecvWorker@918] - Interrupting SendWorker
2016-09-13 05:09:25,907 [myid:0] - WARN [SendWorker:2:QuorumCnxManager$SendWorker@837] - Exception when using channel: for id 2 my id = 0 error = java.net.SocketException: Broken pipe
2016-09-13 05:09:25,907 [myid:0] - INFO [/127.0.0.1:13917:QuorumCnxManager$Listener@638] - Received connection request /127.0.0.1:53030
2016-09-13 05:09:25,908 [myid:0] - WARN [SendWorker:2:QuorumCnxManager$SendWorker@841] - Send worker leaving thread id 2 my id = 0
2016-09-13 05:09:25,908 [myid:0] - WARN [RecvWorker:2:QuorumCnxManager$RecvWorker@918] - Interrupting SendWorker
2016-09-13 05:09:25,909 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@688] - Notification: 2 (message format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 0 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version)
2016-09-13 05:09:25,910 [myid:0] - INFO [WorkerReceiver[myid=0]:FastLeaderElection@688] - Notification: 2 (message format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version)
2016-09-13 05:09:26,022 [myid:] - INFO [main:FourLetterWordMain@85] - connecting to 127.0.0.1 13915
2016-09-13 05:09:26,023 [myid:0] - INFO [NIOServerCxnFactory.AcceptThread:/127.0.0.1:13915:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:38382
2016-09-13 05:09:26,032 [myid:0] - INFO [NIOWorkerThread-2:NIOServerCnxn@485] - Processing stat command from /127.0.0.1:38382
2016-09-13 05:09:26,033 [myid:0] - INFO [NIOWorkerThread-2:NIOServerCnxn@607] - Closed socket connection for client /127.0.0.1:38382 (no session established for client)
2016-09-13 05:09:26,103 [myid:1] - INFO [QuorumPeer[myid=1](plain=/127.0.0.1:13918)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id1,name1=replica.1,name2=LeaderElection]
2016-09-13 05:09:26,104 [myid:1] - INFO [QuorumPeer[myid=1](plain=/127.0.0.1:13918)(secure=disabled):QuorumPeer@1109] - FOLLOWING
2016-09-13 05:09:26,109 [myid:2] - INFO [QuorumPeer[myid=2](plain=/127.0.0.1:13921)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id2,name1=replica.2,name2=LeaderElection]
2016-09-13 05:09:26,110 [myid:1] - INFO [QuorumPeer[myid=1](plain=/127.0.0.1:13918)(secure=disabled):Learner@88] - TCP NoDelay set to: true
2016-09-13 05:09:26,110 [myid:2] - INFO [QuorumPeer[myid=2](plain=/127.0.0.1:13921)(secure=disabled):QuorumPeer@1121] - LEADING
2016-09-13 05:09:26,111 [myid:0] - INFO [QuorumPeer[myid=0](plain=/127.0.0.1:13915)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id0,name1=replica.0,name2=LeaderElection]
2016-09-13 05:09:26,111 [myid:0] - INFO [QuorumPeer[myid=0](plain=/127.0.0.1:13915)(secure=disabled):QuorumPeer@1109] - FOLLOWING
2016-09-13 05:09:26,113 [myid:2] - INFO [QuorumPeer[myid=2](plain=/127.0.0.1:13921)(secure=disabled):Leader@63] - TCP NoDelay set to: true
2016-09-13 05:09:26,113 [myid:2] - INFO [QuorumPeer[myid=2](plain=/127.0.0.1:13921)(secure=disabled):Leader@83] - zookeeper.leader.maxConcurrentSnapshots = 10
2016-09-13 05:09:26,113 [myid:2] - INFO [QuorumPeer[myid=2](plain=/127.0.0.1:13921)(secure=disabled):Leader@85] - zookeeper.leader.maxConcurrentSnapshotTimeout = 5
2016-09-13 05:09:26,122 [myid:0] - INFO [QuorumPeer[myid=0](plain=/127.0.0.1:13915)(secure=disabled):Environment@109] - Server environment:zookeeper.version=3.6.0-SNAPSHOT--1, built on 09/13/2016 05:08 GMT
2016-09-13 05:09:26,122 [myid:0] - INFO [QuorumPeer[myid=0](plain=/127.0.0.1:13915)(secure=disabled):Environment@109] - Server environment:host.name=jenkins-test-5f3
2016-09-13 05:09:26,123 [myid:0] - INFO [QuorumPeer[myid=0](plain=/127.0.0.1:13915)(secure=disabled):Environment@109] - Server environment:java.version=1.7.0_101
2016-09-13 05:09:26,123 [myid:0] - INFO [QuorumPeer[myid=0](plain=/127.0.0.1:13915)(secure=disabled):Environment@109] - Server environment:java.vendor=Oracle Corporation
2016-09-13 05:09:26,123 [myid:0] - INFO [QuorumPeer[myid=0](plain=/127.0.0.1:13915)(secure=disabled):Environment@109] - Server environment:java.home=/usr/lib/jvm/java-7-openjdk-amd64/jre
2016-09-13 05:09:26,123 [myid:0] - INFO [QuorumPeer[myid=0](plain=/127.0.0.1:13915)(secure=disabled):Environment@109] - Server environment:java.class.path=/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/test/classes:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/test/lib/antlr-2.7.7.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/test/lib/antlr4-runtime-4.5.1-1.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/test/lib/checkstyle-6.13.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/test/lib/commons-beanutils-1.9.2.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/test/lib/commons-cli-1.3.1.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/test/lib/commons-collections-3.2.2.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/test/lib/commons-lang3-3.4.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/test/lib/commons-logging-1.1.1.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/test/lib/guava-18.0.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/test/lib/hamcrest-core-1.3.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/test/lib/junit-4.12.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/test/lib/mockito-all-1.8.2.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/classes:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/src/java/lib/ivy-2.4.0.jar:/home/jenkins/tools/ant/latest/lib/ant.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/lib/commons-cli-1.2.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/lib/jackson-core-asl-1.9.11.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/lib/jackson-mapper-asl-1.9.11.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/lib/javacc.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/lib/javax.servlet-api-3.1.0.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/lib/jetty-http-9.2.18.v20160721.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/lib/jetty-io-9.2.18.v20160721.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/lib/jetty-security-9.2.18.v20160721.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/lib/jetty-server-9.2.18.v20160721.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/lib/jetty-servlet-9.2.18.v20160721.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/lib/jetty-util-9.2.18.v20160721.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/lib/jline-2.11.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/lib/log4j-1.2.17.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/lib/netty-3.10.5.Final.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/lib/slf4j-api-1.7.5.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/lib/slf4j-log4j12-1.7.5.jar:/usr/local/asfpackages/ant/apache-ant-1.9.7/lib/ant-launcher.jar:/home/jenkins/tools/ant/latest/lib/ant-junit.jar:/home/jenkins/tools/ant/latest/lib/ant-junit4.jar
2016-09-13 05:09:26,123 [myid:0] - INFO [QuorumPeer[myid=0](plain=/127.0.0.1:13915)(secure=disabled):Environment@109] - Server environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib
2016-09-13 05:09:26,123 [myid:0] - INFO [QuorumPeer[myid=0](plain=/127.0.0.1:13915)(secure=disabled):Environment@109] - Server environment:java.io.tmpdir=/tmp
2016-09-13 05:09:26,124 [myid:0] - INFO [QuorumPeer[myid=0](plain=/127.0.0.1:13915)(secure=disabled):Environment@109] - Server environment:java.compiler=<NA>
2016-09-13 05:09:26,124 [myid:0] - INFO [QuorumPeer[myid=0](plain=/127.0.0.1:13915)(secure=disabled):Environment@109] - Server environment:os.name=Linux
2016-09-13 05:09:26,124 [myid:0] - INFO [QuorumPeer[myid=0](plain=/127.0.0.1:13915)(secure=disabled):Environment@109] - Server environment:os.arch=amd64
2016-09-13 05:09:26,124 [myid:0] - INFO [QuorumPeer[myid=0](plain=/127.0.0.1:13915)(secure=disabled):Environment@109] - Server environment:os.version=3.13.0-30-generic
2016-09-13 05:09:26,124 [myid:0] - INFO [QuorumPeer[myid=0](plain=/127.0.0.1:13915)(secure=disabled):Environment@109] - Server environment:user.name=jenkins
2016-09-13 05:09:26,124 [myid:0] - INFO [QuorumPeer[myid=0](plain=/127.0.0.1:13915)(secure=disabled):Environment@109] - Server environment:user.home=/home/jenkins
2016-09-13 05:09:26,124 [myid:0] - INFO [QuorumPeer[myid=0](plain=/127.0.0.1:13915)(secure=disabled):Environment@109] - Server environment:user.dir=/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/test
2016-09-13 05:09:26,125 [myid:0] - INFO [QuorumPeer[myid=0](plain=/127.0.0.1:13915)(secure=disabled):Environment@109] - Server environment:os.memory.free=49MB
2016-09-13 05:09:26,125 [myid:0] - INFO [QuorumPeer[myid=0](plain=/127.0.0.1:13915)(secure=disabled):Environment@109] - Server environment:os.memory.max=455MB
2016-09-13 05:09:26,125 [myid:0] - INFO [QuorumPeer[myid=0](plain=/127.0.0.1:13915)(secure=disabled):Environment@109] - Server environment:os.memory.total=60MB
2016-09-13 05:09:26,126 [myid:2] - INFO [QuorumPeer[myid=2](plain=/127.0.0.1:13921)(secure=disabled):ZooKeeperServer@889] - minSessionTimeout set to 8000
2016-09-13 05:09:26,127 [myid:2] - INFO [QuorumPeer[myid=2](plain=/127.0.0.1:13921)(secure=disabled):ZooKeeperServer@898] - maxSessionTimeout set to 80000
2016-09-13 05:09:26,127 [myid:2] - INFO [QuorumPeer[myid=2](plain=/127.0.0.1:13921)(secure=disabled):ZooKeeperServer@159] - Created server with tickTime 4000 minSessionTimeout 8000 maxSessionTimeout 80000 datadir /home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/test/tmp/test5455612786130415623.junit.dir/data/version-2 snapdir /home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/test/tmp/test5455612786130415623.junit.dir/data/version-2
2016-09-13 05:09:26,126 [myid:0] - INFO [QuorumPeer[myid=0](plain=/127.0.0.1:13915)(secure=disabled):ZooKeeperServer@889] - minSessionTimeout set to 8000
2016-09-13 05:09:26,129 [myid:0] - INFO [QuorumPeer[myid=0](plain=/127.0.0.1:13915)(secure=disabled):ZooKeeperServer@898] - maxSessionTimeout set to 80000
2016-09-13 05:09:26,129 [myid:0] - INFO [QuorumPeer[myid=0](plain=/127.0.0.1:13915)(secure=disabled):ZooKeeperServer@159] - Created server with tickTime 4000 minSessionTimeout 8000 maxSessionTimeout 80000 datadir /home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/test/tmp/test8397079557861207505.junit.dir/data/version-2 snapdir /home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/test/tmp/test8397079557861207505.junit.dir/data/version-2
2016-09-13 05:09:26,129 [myid:0] - INFO [QuorumPeer[myid=0](plain=/127.0.0.1:13915)(secure=disabled):Follower@66] - FOLLOWING - LEADER ELECTION TOOK - 18 MS
2016-09-13 05:09:26,126 [myid:1] - INFO [QuorumPeer[myid=1](plain=/127.0.0.1:13918)(secure=disabled):ZooKeeperServer@889] - minSessionTimeout set to 8000
2016-09-13 05:09:26,129 [myid:2] - INFO [QuorumPeer[myid=2](plain=/127.0.0.1:13921)(secure=disabled):Leader@412] - LEADING - LEADER ELECTION TOOK - 20 MS
2016-09-13 05:09:26,140 [myid:2] - INFO [QuorumPeer[myid=2](plain=/127.0.0.1:13921)(secure=disabled):FileTxnSnapLog@298] - Snapshotting: 0x0 to /home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/test/tmp/test5455612786130415623.junit.dir/data/version-2/snapshot.0
2016-09-13 05:09:26,137 [myid:1] - INFO [QuorumPeer[myid=1](plain=/127.0.0.1:13918)(secure=disabled):ZooKeeperServer@898] - maxSessionTimeout set to 80000
2016-09-13 05:09:26,141 [myid:1] - INFO [QuorumPeer[myid=1](plain=/127.0.0.1:13918)(secure=disabled):ZooKeeperServer@159] - Created server with tickTime 4000 minSessionTimeout 8000 maxSessionTimeout 80000 datadir /home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/test/tmp/test1768888919429940621.junit.dir/data/version-2 snapdir /home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/test/tmp/test1768888919429940621.junit.dir/data/version-2
2016-09-13 05:09:26,141 [myid:1] - INFO [QuorumPeer[myid=1](plain=/127.0.0.1:13918)(secure=disabled):Follower@66] - FOLLOWING - LEADER ELECTION TOOK - 37 MS
2016-09-13 05:09:26,185 [myid:2] - INFO [LearnerHandler-/127.0.0.1:39078:LearnerHandler@382] - Follower sid: 0 : info : 127.0.0.1:13916:13917:participant;127.0.0.1:13915
2016-09-13 05:09:26,191 [myid:2] - INFO [LearnerHandler-/127.0.0.1:39079:LearnerHandler@382] - Follower sid: 1 : info : 127.0.0.1:13919:13920:participant;127.0.0.1:13918
2016-09-13 05:09:26,198 [myid:2] - INFO [LearnerHandler-/127.0.0.1:39079:LearnerHandler@683] - Synchronizing with Follower sid: 1 maxCommittedLog=0x0 minCommittedLog=0x0 lastProcessedZxid=0x0 peerLastZxid=0x0
2016-09-13 05:09:26,198 [myid:2] - INFO [LearnerHandler-/127.0.0.1:39079:LearnerHandler@727] - Sending DIFF zxid=0x0 for peer sid: 1
2016-09-13 05:09:26,198 [myid:2] - INFO [LearnerHandler-/127.0.0.1:39078:LearnerHandler@683] - Synchronizing with Follower sid: 0 maxCommittedLog=0x0 minCommittedLog=0x0 lastProcessedZxid=0x0 peerLastZxid=0x0
2016-09-13 05:09:26,199 [myid:2] - INFO [LearnerHandler-/127.0.0.1:39078:LearnerHandler@727] - Sending DIFF zxid=0x0 for peer sid: 0
2016-09-13 05:09:26,201 [myid:1] - INFO [QuorumPeer[myid=1](plain=/127.0.0.1:13918)(secure=disabled):Learner@366] - Getting a diff from the leader 0x0
2016-09-13 05:09:26,206 [myid:1] - INFO [QuorumPeer[myid=1](plain=/127.0.0.1:13918)(secure=disabled):Learner@509] - Learner received NEWLEADER message
2016-09-13 05:09:26,209 [myid:1] - INFO [QuorumPeer[myid=1](plain=/127.0.0.1:13918)(secure=disabled):FileTxnSnapLog@298] - Snapshotting: 0x0 to /home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/test/tmp/test1768888919429940621.junit.dir/data/version-2/snapshot.0
2016-09-13 05:09:26,210 [myid:0] - INFO [QuorumPeer[myid=0](plain=/127.0.0.1:13915)(secure=disabled):Learner@366] - Getting a diff from the leader 0x0
2016-09-13 05:09:26,211 [myid:0] - INFO [QuorumPeer[myid=0](plain=/127.0.0.1:13915)(secure=disabled):Learner@509] - Learner received NEWLEADER message
2016-09-13 05:09:26,213 [myid:2] - INFO [QuorumPeer[myid=2](plain=/127.0.0.1:13921)(secure=disabled):Leader@1255] - Have quorum of supporters, sids: [ [1, 2],[1, 2] ]; starting up and setting last processed zxid: 0x100000000
2016-09-13 05:09:26,215 [myid:0] - INFO [QuorumPeer[myid=0](plain=/127.0.0.1:13915)(secure=disabled):FileTxnSnapLog@298] - Snapshotting: 0x0 to /home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/test/tmp/test8397079557861207505.junit.dir/data/version-2/snapshot.0
2016-09-13 05:09:26,238 [myid:2] - INFO [QuorumPeer[myid=2](plain=/127.0.0.1:13921)(secure=disabled):CommitProcessor@318] - Configuring CommitProcessor with 4 worker threads.
2016-09-13 05:09:26,252 [myid:2] - INFO [QuorumPeer[myid=2](plain=/127.0.0.1:13921)(secure=disabled):ContainerManager@64] - Using checkIntervalMs=60000 maxPerMinute=10000
2016-09-13 05:09:26,254 [myid:0] - INFO [QuorumPeer[myid=0](plain=/127.0.0.1:13915)(secure=disabled):Learner@493] - Learner received UPTODATE message
2016-09-13 05:09:26,254 [myid:1] - INFO [QuorumPeer[myid=1](plain=/127.0.0.1:13918)(secure=disabled):Learner@493] - Learner received UPTODATE message
2016-09-13 05:09:26,266 [myid:1] - INFO [QuorumPeer[myid=1](plain=/127.0.0.1:13918)(secure=disabled):CommitProcessor@318] - Configuring CommitProcessor with 4 worker threads.
2016-09-13 05:09:26,272 [myid:0] - INFO [QuorumPeer[myid=0](plain=/127.0.0.1:13915)(secure=disabled):CommitProcessor@318] - Configuring CommitProcessor with 4 worker threads.
2016-09-13 05:09:26,283 [myid:] - INFO [main:FourLetterWordMain@85] - connecting to 127.0.0.1 13915
2016-09-13 05:09:26,284 [myid:0] - INFO [NIOServerCxnFactory.AcceptThread:/127.0.0.1:13915:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:38385
2016-09-13 05:09:26,285 [myid:0] - INFO [NIOWorkerThread-3:NIOServerCnxn@485] - Processing stat command from /127.0.0.1:38385
2016-09-13 05:09:26,286 [myid:0] - INFO [NIOWorkerThread-3:StatCommand@49] - Stat command output
2016-09-13 05:09:26,287 [myid:0] - INFO [NIOWorkerThread-3:NIOServerCnxn@607] - Closed socket connection for client /127.0.0.1:38385 (no session established for client)
2016-09-13 05:09:26,288 [myid:] - INFO [main:FourLetterWordMain@85] - connecting to 127.0.0.1 13918
2016-09-13 05:09:26,288 [myid:1] - INFO [NIOServerCxnFactory.AcceptThread:/127.0.0.1:13918:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:55309
2016-09-13 05:09:26,291 [myid:1] - INFO [NIOWorkerThread-1:NIOServerCnxn@485] - Processing stat command from /127.0.0.1:55309
2016-09-13 05:09:26,291 [myid:1] - INFO [NIOWorkerThread-1:StatCommand@49] - Stat command output
2016-09-13 05:09:26,292 [myid:1] - INFO [NIOWorkerThread-1:NIOServerCnxn@607] - Closed socket connection for client /127.0.0.1:55309 (no session established for client)
2016-09-13 05:09:26,292 [myid:] - INFO [main:FourLetterWordMain@85] - connecting to 127.0.0.1 13921
2016-09-13 05:09:26,293 [myid:2] - INFO [NIOServerCxnFactory.AcceptThread:/127.0.0.1:13921:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:45002
2016-09-13 05:09:26,297 [myid:2] - INFO [NIOWorkerThread-1:NIOServerCnxn@485] - Processing stat command from /127.0.0.1:45002
2016-09-13 05:09:26,297 [myid:2] - INFO [NIOWorkerThread-1:StatCommand@49] - Stat command output
2016-09-13 05:09:26,298 [myid:2] - INFO [NIOWorkerThread-1:NIOServerCnxn@607] - Closed socket connection for client /127.0.0.1:45002 (no session established for client)
2016-09-13 05:09:26,304 [myid:] - INFO [main:Environment@109] - Client environment:zookeeper.version=3.6.0-SNAPSHOT--1, built on 09/13/2016 05:08 GMT
2016-09-13 05:09:26,304 [myid:] - INFO [main:Environment@109] - Client environment:host.name=jenkins-test-5f3
2016-09-13 05:09:26,304 [myid:] - INFO [main:Environment@109] - Client environment:java.version=1.7.0_101
2016-09-13 05:09:26,305 [myid:] - INFO [main:Environment@109] - Client environment:java.vendor=Oracle Corporation
2016-09-13 05:09:26,305 [myid:] - INFO [main:Environment@109] - Client environment:java.home=/usr/lib/jvm/java-7-openjdk-amd64/jre
2016-09-13 05:09:26,305 [myid:] - INFO [main:Environment@109] - Client environment:java.class.path=/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/test/classes:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/test/lib/antlr-2.7.7.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/test/lib/antlr4-runtime-4.5.1-1.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/test/lib/checkstyle-6.13.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/test/lib/commons-beanutils-1.9.2.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/test/lib/commons-cli-1.3.1.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/test/lib/commons-collections-3.2.2.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/test/lib/commons-lang3-3.4.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/test/lib/commons-logging-1.1.1.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/test/lib/guava-18.0.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/test/lib/hamcrest-core-1.3.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/test/lib/junit-4.12.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/test/lib/mockito-all-1.8.2.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/classes:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/src/java/lib/ivy-2.4.0.jar:/home/jenkins/tools/ant/latest/lib/ant.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/lib/commons-cli-1.2.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/lib/jackson-core-asl-1.9.11.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/lib/jackson-mapper-asl-1.9.11.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/lib/javacc.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/lib/javax.servlet-api-3.1.0.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/lib/jetty-http-9.2.18.v20160721.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/lib/jetty-io-9.2.18.v20160721.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/lib/jetty-security-9.2.18.v20160721.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/lib/jetty-server-9.2.18.v20160721.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/lib/jetty-servlet-9.2.18.v20160721.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/lib/jetty-util-9.2.18.v20160721.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/lib/jline-2.11.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/lib/log4j-1.2.17.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/lib/netty-3.10.5.Final.
...[truncated 31909 chars]...
3] - WARN [RecvWorker:1:QuorumCnxManager$RecvWorker@915] - Connection broken for id 1, my id = 3, error =
java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:392)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:900)
2016-09-13 05:09:26,894 [myid:3] - WARN [RecvWorker:1:QuorumCnxManager$RecvWorker@918] - Interrupting SendWorker
2016-09-13 05:09:26,894 [myid:3] - WARN [SendWorker:1:QuorumCnxManager$SendWorker@832] - Interrupted while waiting for message on queue
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:982)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:63)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:820)
2016-09-13 05:09:26,894 [myid:3] - WARN [SendWorker:1:QuorumCnxManager$SendWorker@841] - Send worker leaving thread id 1 my id = 3
2016-09-13 05:09:26,895 [myid:1] - WARN [RecvWorker:3:QuorumCnxManager$RecvWorker@915] - Connection broken for id 3, my id = 1, error =
java.net.SocketException: Socket closed
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at java.net.SocketInputStream.read(SocketInputStream.java:210)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:900)
2016-09-13 05:09:26,895 [myid:1] - WARN [RecvWorker:3:QuorumCnxManager$RecvWorker@918] - Interrupting SendWorker
2016-09-13 05:09:26,895 [myid:1] - WARN [SendWorker:3:QuorumCnxManager$SendWorker@832] - Interrupted while waiting for message on queue
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:982)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:63)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:820)
2016-09-13 05:09:26,896 [myid:1] - WARN [SendWorker:3:QuorumCnxManager$SendWorker@841] - Send worker leaving thread id 3 my id = 1
2016-09-13 05:09:26,905 [myid:0] - WARN [SendWorker:1:QuorumCnxManager$SendWorker@841] - Send worker leaving thread id 1 my id = 0
2016-09-13 05:09:26,905 [myid:0] - WARN [RecvWorker:3:QuorumCnxManager$RecvWorker@915] - Connection broken for id 3, my id = 0, error =
java.net.SocketException: Socket closed
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at java.net.SocketInputStream.read(SocketInputStream.java:210)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:900)
2016-09-13 05:09:26,906 [myid:0] - WARN [RecvWorker:3:QuorumCnxManager$RecvWorker@918] - Interrupting SendWorker
2016-09-13 05:09:26,905 [myid:3] - WARN [RecvWorker:0:QuorumCnxManager$RecvWorker@915] - Connection broken for id 0, my id = 3, error =
java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:392)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:900)
2016-09-13 05:09:26,906 [myid:3] - WARN [RecvWorker:0:QuorumCnxManager$RecvWorker@918] - Interrupting SendWorker
2016-09-13 05:09:26,905 [myid:0] - WARN [SendWorker:3:QuorumCnxManager$SendWorker@832] - Interrupted while waiting for message on queue
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:982)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:63)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:820)
2016-09-13 05:09:26,913 [myid:3] - WARN [QuorumPeer[myid=3](plain=/0:0:0:0:0:0:0:0:13924)(secure=disabled):Learner@417] - Got zxid 0x100000002 expected 0x1
2016-09-13 05:09:26,917 [myid:3] - WARN [SendWorker:0:QuorumCnxManager$SendWorker@832] - Interrupted while waiting for message on queue
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:982)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:63)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:820)
2016-09-13 05:09:26,918 [myid:3] - WARN [SendWorker:0:QuorumCnxManager$SendWorker@841] - Send worker leaving thread id 0 my id = 3
2016-09-13 05:09:26,917 [myid:0] - WARN [SendWorker:3:QuorumCnxManager$SendWorker@841] - Send worker leaving thread id 3 my id = 0
2016-09-13 05:09:26,917 [myid:1] - INFO [QuorumPeerListener:QuorumCnxManager$Listener@632] - My election bind port: /127.0.0.1:13920
2016-09-13 05:09:26,919 [myid:3] - INFO [QuorumPeer[myid=3](plain=/0:0:0:0:0:0:0:0:13924)(secure=disabled):QuorumPeer@1481] - writeToDisk == true but configFilename == null
2016-09-13 05:09:26,920 [myid:3] - WARN [QuorumPeer[myid=3](plain=/0:0:0:0:0:0:0:0:13924)(secure=disabled):QuorumPeer@1417] - Restarting Leader Election
2016-09-13 05:09:26,920 [myid:3] - INFO [/127.0.0.1:13926:QuorumCnxManager$Listener@661] - Leaving listener
2016-09-13 05:09:26,922 [myid:0] - INFO [QuorumPeerListener:QuorumCnxManager$Listener@632] - My election bind port: /127.0.0.1:13917
2016-09-13 05:09:26,936 [myid:3] - INFO [QuorumPeer[myid=3](plain=/0:0:0:0:0:0:0:0:13924)(secure=disabled):Learner@493] - Learner received UPTODATE message
2016-09-13 05:09:26,936 [myid:3] - INFO [QuorumPeerListener:QuorumCnxManager$Listener@632] - My election bind port: /127.0.0.1:13926
2016-09-13 05:09:27,036 [myid:3] - INFO [QuorumPeer[myid=3](plain=/0:0:0:0:0:0:0:0:13924)(secure=disabled):CommitProcessor@318] - Configuring CommitProcessor with 4 worker threads.
2016-09-13 05:09:27,061 [myid:3] - INFO [SyncThread:3:FileTxnLog@204] - Creating new log file: log.100000002
2016-09-13 05:09:28,635 [myid:127.0.0.1:13924] - INFO [main-SendThread(127.0.0.1:13924):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:13924. Will not attempt to authenticate using SASL (unknown error)
2016-09-13 05:09:28,635 [myid:127.0.0.1:13924] - INFO [main-SendThread(127.0.0.1:13924):ClientCnxn$SendThread@948] - Socket connection established, initiating session, client: /127.0.0.1:50397, server: 127.0.0.1/127.0.0.1:13924
2016-09-13 05:09:28,636 [myid:3] - INFO [NIOServerCxnFactory.AcceptThread:0.0.0.0/0.0.0.0:13924:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:50397
2016-09-13 05:09:28,651 [myid:3] - INFO [NIOWorkerThread-2:ZooKeeperServer@995] - Client attempting to establish new session at /127.0.0.1:50397
2016-09-13 05:09:28,652 [myid:3] - WARN [QuorumPeer[myid=3](plain=/0:0:0:0:0:0:0:0:13924)(secure=disabled):Follower@122] - Got zxid 0x100000003 expected 0x1
2016-09-13 05:09:28,654 [myid:3] - INFO [CommitProcWorkThread-1:ZooKeeperServer@709] - Established session 0x3000048422c0000 with negotiated timeout 30000 for client /127.0.0.1:50397
2016-09-13 05:09:28,661 [myid:127.0.0.1:13924] - INFO [main-SendThread(127.0.0.1:13924):ClientCnxn$SendThread@1381] - Session establishment complete on server 127.0.0.1/127.0.0.1:13924, sessionid = 0x3000048422c0000, negotiated timeout = 30000
2016-09-13 05:09:28,690 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@98] - TEST METHOD FAILED testDuringLeaderSync
java.lang.AssertionError: zoo.cfg.dynamic.next is not deleted.
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.junit.Assert.assertFalse(Assert.java:64)
at org.apache.zookeeper.server.quorum.ReconfigDuringLeaderSyncTest.testDuringLeaderSync(ReconfigDuringLeaderSyncTest.java:155)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:79)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.TestWatchman$1.evaluate(TestWatchman.java:53)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:38)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:535)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1182)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:1033)
2016-09-13 05:09:28,692 [myid:] - INFO [main:QuorumBase@394] - Shutting down quorum peer QuorumPeer[myid=0](plain=/127.0.0.1:13915)(secure=disabled)
2016-09-13 05:09:28,692 [myid:] - INFO [main:Follower@198] - shutdown called
java.lang.Exception: shutdown Follower
at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:198)
at org.apache.zookeeper.server.quorum.QuorumPeer.shutdown(QuorumPeer.java:1184)
at org.apache.zookeeper.test.QuorumBase.shutdown(QuorumBase.java:395)
at org.apache.zookeeper.server.quorum.QuorumPeerTestBase$TestQPMain.shutdown(QuorumPeerTestBase.java:60)
at org.apache.zookeeper.server.quorum.QuorumPeerTestBase$MainThread.shutdown(QuorumPeerTestBase.java:257)
at org.apache.zookeeper.server.quorum.ReconfigDuringLeaderSyncTest.tearDown(ReconfigDuringLeaderSyncTest.java:189)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33)
at org.junit.rules.TestWatchman$1.evaluate(TestWatchman.java:53)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:38)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:535)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1182)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:1033)
2016-09-13 05:09:28,693 [myid:] - INFO [main:MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id0,name1=replica.0,name2=Follower,name3=Connections,name4=127.0.0.1,name5=0x483fd50000]
2016-09-13 05:09:28,694 [myid:] - INFO [main:NIOServerCnxn@607] - Closed socket connection for client /127.0.0.1:38389 which had sessionid 0x483fd50000
2016-09-13 05:09:28,694 [myid:] - INFO [main:LearnerZooKeeperServer@165] - Shutting down
2016-09-13 05:09:28,694 [myid:] - INFO [main:ZooKeeperServer@529] - shutting down
2016-09-13 05:09:28,694 [myid:] - INFO [main:FollowerRequestProcessor@138] - Shutting down
2016-09-13 05:09:28,695 [myid:127.0.0.1:13915] - INFO [main-SendThread(127.0.0.1:13915):ClientCnxn$SendThread@1231] - Unable to read additional data from server sessionid 0x483fd50000, likely server has closed socket, closing socket connection and attempting reconnect
2016-09-13 05:09:28,695 [myid:] - INFO [main:CommitProcessor@414] - Shutting down
2016-09-13 05:09:28,695 [myid:0] - INFO [FollowerRequestProcessor:0:FollowerRequestProcessor@109] - FollowerRequestProcessor exited loop!
2016-09-13 05:09:28,696 [myid:0] - INFO [CommitProcessor:0:CommitProcessor@299] - CommitProcessor exited loop!
2016-09-13 05:09:28,696 [myid:] - INFO [main:FinalRequestProcessor@479] - shutdown of request processor complete
2016-09-13 05:09:28,697 [myid:] - INFO [main:MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id0,name1=replica.0,name2=Follower,name3=InMemoryDataTree]
2016-09-13 05:09:28,697 [myid:] - INFO [main:SyncRequestProcessor@191] - Shutting down
2016-09-13 05:09:28,697 [myid:0] - INFO [SyncThread:0:SyncRequestProcessor@169] - SyncRequestProcessor exited!
2016-09-13 05:09:28,698 [myid:0] - INFO [ConnnectionExpirer:NIOServerCnxnFactory$ConnectionExpirerThread@583] - ConnnectionExpirerThread interrupted
2016-09-13 05:09:28,698 [myid:0] - INFO [NIOServerCxnFactory.AcceptThread:/127.0.0.1:13915:NIOServerCnxnFactory$AcceptThread@219] - accept thread exitted run method
2016-09-13 05:09:28,699 [myid:0] - INFO [NIOServerCxnFactory.SelectorThread-0:NIOServerCnxnFactory$SelectorThread@420] - selector thread exitted run method
2016-09-13 05:09:28,699 [myid:0] - INFO [/127.0.0.1:13917:QuorumCnxManager$Listener@661] - Leaving listener
2016-09-13 05:09:28,700 [myid:] - INFO [main:QuorumBase@398] - Shutting down leader election QuorumPeer[myid=0](plain=/127.0.0.1:13915)(secure=disabled)
2016-09-13 05:09:28,700 [myid:] - INFO [main:QuorumBase@403] - Waiting for QuorumPeer[myid=0](plain=/127.0.0.1:13915)(secure=disabled) to exit thread
2016-09-13 05:09:29,738 [myid:0] - INFO [WorkerReceiver[myid=0]:FastLeaderElection$Messenger$WorkerReceiver@440] - WorkerReceiver is down
2016-09-13 05:09:29,738 [myid:0] - INFO [WorkerSender[myid=0]:FastLeaderElection$Messenger$WorkerSender@470] - WorkerSender is down
2016-09-13 05:09:29,776 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection$Messenger$WorkerReceiver@440] - WorkerReceiver is down
2016-09-13 05:09:29,785 [myid:3] - INFO [WorkerSender[myid=3]:FastLeaderElection$Messenger$WorkerSender@470] - WorkerSender is down
2016-09-13 05:09:29,785 [myid:1] - INFO [WorkerSender[myid=1]:FastLeaderElection$Messenger$WorkerSender@470] - WorkerSender is down
2016-09-13 05:09:29,789 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@440] - WorkerReceiver is down
2016-09-13 05:09:29,790 [myid:2] - INFO [WorkerSender[myid=2]:FastLeaderElection$Messenger$WorkerSender@470] - WorkerSender is down
2016-09-13 05:09:29,792 [myid:3] - INFO [WorkerReceiver[myid=3]:FastLeaderElection$Messenger$WorkerReceiver@440] - WorkerReceiver is down
2016-09-13 05:09:29,922 [myid:0] - INFO [WorkerSender[myid=0]:FastLeaderElection$Messenger$WorkerSender@470] - WorkerSender is down
2016-09-13 05:09:29,923 [myid:0] - INFO [WorkerReceiver[myid=0]:FastLeaderElection$Messenger$WorkerReceiver@440] - WorkerReceiver is down
2016-09-13 05:09:30,255 [myid:0] - INFO [QuorumPeer[myid=0](plain=/127.0.0.1:13915)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id0,name1=replica.0,name2=Follower]
2016-09-13 05:09:30,256 [myid:0] - INFO [QuorumPeer[myid=0](plain=/127.0.0.1:13915)(secure=disabled):Follower@198] - shutdown called
java.lang.Exception: shutdown Follower
at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:198)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1115)
2016-09-13 05:09:30,256 [myid:0] - WARN [QuorumPeer[myid=0](plain=/127.0.0.1:13915)(secure=disabled):QuorumPeer@1158] - PeerState set to LOOKING
2016-09-13 05:09:30,256 [myid:0] - WARN [QuorumPeer[myid=0](plain=/127.0.0.1:13915)(secure=disabled):QuorumPeer@1140] - QuorumPeer main thread exited
2016-09-13 05:09:30,256 [myid:0] - INFO [QuorumPeer[myid=0](plain=/127.0.0.1:13915)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id0]
2016-09-13 05:09:30,256 [myid:0] - INFO [QuorumPeer[myid=0](plain=/127.0.0.1:13915)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id0,name1=replica.0]
2016-09-13 05:09:30,256 [myid:0] - INFO [QuorumPeer[myid=0](plain=/127.0.0.1:13915)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id0,name1=replica.1]
2016-09-13 05:09:30,257 [myid:0] - INFO [QuorumPeer[myid=0](plain=/127.0.0.1:13915)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id0,name1=replica.2]
2016-09-13 05:09:30,257 [myid:0] - INFO [QuorumPeer[myid=0](plain=/127.0.0.1:13915)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id0,name1=replica.3]
2016-09-13 05:09:30,258 [myid:] - INFO [main:QuorumBase@394] - Shutting down quorum peer QuorumPeer[myid=1](plain=/127.0.0.1:13918)(secure=disabled)
2016-09-13 05:09:30,258 [myid:] - INFO [main:Follower@198] - shutdown called
java.lang.Exception: shutdown Follower
at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:198)
at org.apache.zookeeper.server.quorum.QuorumPeer.shutdown(QuorumPeer.java:1184)
at org.apache.zookeeper.test.QuorumBase.shutdown(QuorumBase.java:395)
at org.apache.zookeeper.server.quorum.QuorumPeerTestBase$TestQPMain.shutdown(QuorumPeerTestBase.java:60)
at org.apache.zookeeper.server.quorum.QuorumPeerTestBase$MainThread.shutdown(QuorumPeerTestBase.java:257)
at org.apache.zookeeper.server.quorum.ReconfigDuringLeaderSyncTest.tearDown(ReconfigDuringLeaderSyncTest.java:189)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33)
at org.junit.rules.TestWatchman$1.evaluate(TestWatchman.java:53)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:38)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:535)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1182)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:1033)
2016-09-13 05:09:30,259 [myid:] - INFO [main:LearnerZooKeeperServer@165] - Shutting down
2016-09-13 05:09:30,259 [myid:] - INFO [main:ZooKeeperServer@529] - shutting down
2016-09-13 05:09:30,259 [myid:] - INFO [main:FollowerRequestProcessor@138] - Shutting down
2016-09-13 05:09:30,259 [myid:] - INFO [main:CommitProcessor@414] - Shutting down
2016-09-13 05:09:30,259 [myid:1] - INFO [FollowerRequestProcessor:1:FollowerRequestProcessor@109] - FollowerRequestProcessor exited loop!
2016-09-13 05:09:30,259 [myid:] - INFO [main:FinalRequestProcessor@479] - shutdown of request processor complete
2016-09-13 05:09:30,259 [myid:1] - INFO [CommitProcessor:1:CommitProcessor@299] - CommitProcessor exited loop!
2016-09-13 05:09:30,260 [myid:] - INFO [main:MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id1,name1=replica.1,name2=Follower,name3=InMemoryDataTree]
2016-09-13 05:09:30,260 [myid:] - INFO [main:SyncRequestProcessor@191] - Shutting down
2016-09-13 05:09:30,260 [myid:1] - INFO [SyncThread:1:SyncRequestProcessor@169] - SyncRequestProcessor exited!
2016-09-13 05:09:30,261 [myid:1] - INFO [ConnnectionExpirer:NIOServerCnxnFactory$ConnectionExpirerThread@583] - ConnnectionExpirerThread interrupted
2016-09-13 05:09:30,261 [myid:1] - INFO [NIOServerCxnFactory.AcceptThread:/127.0.0.1:13918:NIOServerCnxnFactory$AcceptThread@219] - accept thread exitted run method
2016-09-13 05:09:30,261 [myid:1] - INFO [NIOServerCxnFactory.SelectorThread-0:NIOServerCnxnFactory$SelectorThread@420] - selector thread exitted run method
2016-09-13 05:09:30,261 [myid:1] - INFO [/127.0.0.1:13920:QuorumCnxManager$Listener@661] - Leaving listener
2016-09-13 05:09:30,261 [myid:] - INFO [main:QuorumBase@398] - Shutting down leader election QuorumPeer[myid=1](plain=/127.0.0.1:13918)(secure=disabled)
2016-09-13 05:09:30,262 [myid:] - INFO [main:QuorumBase@403] - Waiting for QuorumPeer[myid=1](plain=/127.0.0.1:13918)(secure=disabled) to exit thread
2016-09-13 05:09:30,624 [myid:127.0.0.1:13915] - INFO [main-SendThread(127.0.0.1:13915):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:13915. Will not attempt to authenticate using SASL (unknown error)
2016-09-13 05:09:30,624 [myid:127.0.0.1:13915] - WARN [main-SendThread(127.0.0.1:13915):ClientCnxn$SendThread@1235] - Session 0x483fd50000 for server 127.0.0.1/127.0.0.1:13915, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:744)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:357)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1214)
2016-09-13 05:09:31,733 [myid:127.0.0.1:13915] - INFO [main-SendThread(127.0.0.1:13915):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:13915. Will not attempt to authenticate using SASL (unknown error)
2016-09-13 05:09:31,734 [myid:127.0.0.1:13915] - WARN [main-SendThread(127.0.0.1:13915):ClientCnxn$SendThread@1235] - Session 0x483fd50000 for server 127.0.0.1/127.0.0.1:13915, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:744)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:357)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1214)
2016-09-13 05:09:32,255 [myid:1] - INFO [QuorumPeer[myid=1](plain=/127.0.0.1:13918)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id1,name1=replica.1,name2=Follower]
2016-09-13 05:09:32,256 [myid:1] - INFO [QuorumPeer[myid=1](plain=/127.0.0.1:13918)(secure=disabled):Follower@198] - shutdown called
java.lang.Exception: shutdown Follower
at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:198)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1115)
2016-09-13 05:09:32,256 [myid:1] - WARN [QuorumPeer[myid=1](plain=/127.0.0.1:13918)(secure=disabled):QuorumPeer@1158] - PeerState set to LOOKING
2016-09-13 05:09:32,256 [myid:1] - WARN [QuorumPeer[myid=1](plain=/127.0.0.1:13918)(secure=disabled):QuorumPeer@1140] - QuorumPeer main thread exited
2016-09-13 05:09:32,256 [myid:1] - INFO [QuorumPeer[myid=1](plain=/127.0.0.1:13918)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id1]
2016-09-13 05:09:32,257 [myid:1] - INFO [QuorumPeer[myid=1](plain=/127.0.0.1:13918)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id1,name1=replica.1]
2016-09-13 05:09:32,257 [myid:1] - INFO [QuorumPeer[myid=1](plain=/127.0.0.1:13918)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id1,name1=replica.0]
2016-09-13 05:09:32,257 [myid:1] - INFO [QuorumPeer[myid=1](plain=/127.0.0.1:13918)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id1,name1=replica.2]
2016-09-13 05:09:32,257 [myid:1] - INFO [QuorumPeer[myid=1](plain=/127.0.0.1:13918)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id1,name1=replica.3]
2016-09-13 05:09:32,257 [myid:] - INFO [main:QuorumBase@394] - Shutting down quorum peer QuorumPeer[myid=2](plain=/127.0.0.1:13921)(secure=disabled)
2016-09-13 05:09:32,258 [myid:] - INFO [main:Leader@623] - Shutting down
2016-09-13 05:09:32,258 [myid:] - INFO [main:Leader@629] - Shutdown called
java.lang.Exception: shutdown Leader! reason: quorum Peer shutdown
at org.apache.zookeeper.server.quorum.Leader.shutdown(Leader.java:629)
at org.apache.zookeeper.server.quorum.QuorumPeer.shutdown(QuorumPeer.java:1181)
at org.apache.zookeeper.test.QuorumBase.shutdown(QuorumBase.java:395)
at org.apache.zookeeper.server.quorum.QuorumPeerTestBase$TestQPMain.shutdown(QuorumPeerTestBase.java:60)
at org.apache.zookeeper.server.quorum.QuorumPeerTestBase$MainThread.shutdown(QuorumPeerTestBase.java:257)
at org.apache.zookeeper.server.quorum.ReconfigDuringLeaderSyncTest.tearDown(ReconfigDuringLeaderSyncTest.java:189)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33)
at org.junit.rules.TestWatchman$1.evaluate(TestWatchman.java:53)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:38)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:535)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1182)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:1033)
2016-09-13 05:09:32,259 [myid:] - INFO [main:ZooKeeperServer@529] - shutting down
2016-09-13 05:09:32,259 [myid:] - INFO [main:SessionTrackerImpl@232] - Shutting down
2016-09-13 05:09:32,259 [myid:] - INFO [main:LeaderRequestProcessor@77] - Shutting down
2016-09-13 05:09:32,259 [myid:] - INFO [main:PrepRequestProcessor@965] - Shutting down
2016-09-13 05:09:32,259 [myid:] - INFO [main:ProposalRequestProcessor@88] - Shutting down
2016-09-13 05:09:32,259 [myid:2] - INFO [ProcessThread(sid:2 cport:-1)::PrepRequestProcessor@154] - PrepRequestProcessor exited loop!
2016-09-13 05:09:32,259 [myid:] - INFO [main:CommitProcessor@414] - Shutting down
2016-09-13 05:09:32,260 [myid:2] - INFO [CommitProcessor:2:CommitProcessor@299] - CommitProcessor exited loop!
2016-09-13 05:09:32,260 [myid:] - INFO [main:Leader$ToBeAppliedRequestProcessor@924] - Shutting down
2016-09-13 05:09:32,260 [myid:] - INFO [main:FinalRequestProcessor@479] - shutdown of request processor complete
2016-09-13 05:09:32,261 [myid:] - INFO [main:SyncRequestProcessor@191] - Shutting down
2016-09-13 05:09:32,262 [myid:2] - INFO [SyncThread:2:SyncRequestProcessor@169] - SyncRequestProcessor exited!
2016-09-13 05:09:32,262 [myid:2] - INFO [LearnerCnxAcceptor-/127.0.0.1:13922:Leader$LearnerCnxAcceptor@373] - exception while shutting down acceptor: java.net.SocketException: Socket closed
2016-09-13 05:09:32,263 [myid:] - INFO [main:MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id2,name1=replica.2,name2=Leader,name3=InMemoryDataTree]
2016-09-13 05:09:32,264 [myid:2] - WARN [LearnerHandler-/127.0.0.1:39078:LearnerHandler@619] - ******* GOODBYE /127.0.0.1:39078 ********
2016-09-13 05:09:32,265 [myid:3] - WARN [QuorumPeer[myid=3](plain=/0:0:0:0:0:0:0:0:13924)(secure=disabled):Follower@93] - Exception when following the leader
java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:392)
at org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63)
at org.apache.zookeeper.server.quorum.QuorumPacket.deserialize(QuorumPacket.java:83)
at org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:99)
at org.apache.zookeeper.server.quorum.Learner.readPacket(Learner.java:155)
at org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:89)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1111)
2016-09-13 05:09:32,265 [myid:2] - WARN [LearnerHandler-/127.0.0.1:39079:LearnerHandler@619] - ******* GOODBYE /127.0.0.1:39079 ********
2016-09-13 05:09:32,265 [myid:2] - WARN [LearnerHandler-/127.0.0.1:39104:LearnerHandler@619] - ******* GOODBYE /127.0.0.1:39104 ********
2016-09-13 05:09:32,265 [myid:3] - INFO [QuorumPeer[myid=3](plain=/0:0:0:0:0:0:0:0:13924)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id3,name1=replica.3,name2=Follower]
2016-09-13 05:09:32,265 [myid:2] - WARN [LearnerHandler-/127.0.0.1:39104:LearnerHandler@903] - Ignoring unexpected exception
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1219)
at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:340)
at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:338)
at org.apache.zookeeper.server.quorum.LearnerHandler.shutdown(LearnerHandler.java:901)
at org.apache.zookeeper.server.quorum.LearnerHandler.run(LearnerHandler.java:622)
2016-09-13 05:09:32,265 [myid:3] - INFO [QuorumPeer[myid=3](plain=/0:0:0:0:0:0:0:0:13924)(secure=disabled):Follower@198] - shutdown called
java.lang.Exception: shutdown Follower
at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:198)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1115)
2016-09-13 05:09:32,266 [myid:2] - INFO [NIOServerCxnFactory.AcceptThread:/127.0.0.1:13921:NIOServerCnxnFactory$AcceptThread@219] - accept thread exitted run method
2016-09-13 05:09:32,266 [myid:3] - INFO [QuorumPeer[myid=3](plain=/0:0:0:0:0:0:0:0:13924)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id3,name1=replica.3,name2=Follower,name3=Connections,name4=127.0.0.1,name5=0x3000048422c0000]
2016-09-13 05:09:32,266 [myid:2] - INFO [NIOServerCxnFactory.SelectorThread-0:NIOServerCnxnFactory$SelectorThread@420] - selector thread exitted run method
2016-09-13 05:09:32,266 [myid:2] - INFO [ConnnectionExpirer:NIOServerCnxnFactory$ConnectionExpirerThread@583] - ConnnectionExpirerThread interrupted
2016-09-13 05:09:32,266 [myid:3] - INFO [QuorumPeer[myid=3](plain=/0:0:0:0:0:0:0:0:13924)(secure=disabled):NIOServerCnxn@607] - Closed socket connection for client /127.0.0.1:50397 which had sessionid 0x3000048422c0000
2016-09-13 05:09:32,267 [myid:3] - INFO [QuorumPeer[myid=3](plain=/0:0:0:0:0:0:0:0:13924)(secure=disabled):LearnerZooKeeperServer@165] - Shutting down
2016-09-13 05:09:32,267 [myid:3] - INFO [QuorumPeer[myid=3](plain=/0:0:0:0:0:0:0:0:13924)(secure=disabled):ZooKeeperServer@529] - shutting down
2016-09-13 05:09:32,267 [myid:3] - INFO [QuorumPeer[myid=3](plain=/0:0:0:0:0:0:0:0:13924)(secure=disabled):FollowerRequestProcessor@138] - Shutting down
2016-09-13 05:09:32,267 [myid:3] - INFO [QuorumPeer[myid=3](plain=/0:0:0:0:0:0:0:0:13924)(secure=disabled):CommitProcessor@414] - Shutting down
2016-09-13 05:09:32,267 [myid:127.0.0.1:13924] - INFO [main-SendThread(127.0.0.1:13924):ClientCnxn$SendThread@1231] - Unable to read additional data from server sessionid 0x3000048422c0000, likely server has closed socket, closing socket connection and attempting reconnect
2016-09-13 05:09:32,268 [myid:3] - INFO [FollowerRequestProcessor:3:FollowerRequestProcessor@109] - FollowerRequestProcessor exited loop!
2016-09-13 05:09:32,268 [myid:3] - INFO [CommitProcessor:3:CommitProcessor@299] - CommitProcessor exited loop!
2016-09-13 05:09:32,268 [myid:3] - INFO [QuorumPeer[myid=3](plain=/0:0:0:0:0:0:0:0:13924)(secure=disabled):FinalRequestProcessor@479] - shutdown of request processor complete
2016-09-13 05:09:32,269 [myid:3] - INFO [QuorumPeer[myid=3](plain=/0:0:0:0:0:0:0:0:13924)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id3,name1=replica.3,name2=Follower,name3=InMemoryDataTree]
2016-09-13 05:09:32,269 [myid:3] - INFO [QuorumPeer[myid=3](plain=/0:0:0:0:0:0:0:0:13924)(secure=disabled):SyncRequestProcessor@191] - Shutting down
2016-09-13 05:09:32,269 [myid:3] - INFO [SyncThread:3:SyncRequestProcessor@169] - SyncRequestProcessor exited!
2016-09-13 05:09:32,269 [myid:3] - WARN [QuorumPeer[myid=3](plain=/0:0:0:0:0:0:0:0:13924)(secure=disabled):QuorumPeer@1158] - PeerState set to LOOKING
2016-09-13 05:09:32,269 [myid:3] - INFO [QuorumPeer[myid=3](plain=/0:0:0:0:0:0:0:0:13924)(secure=disabled):QuorumPeer@1033] - LOOKING
2016-09-13 05:09:32,270 [myid:3] - INFO [QuorumPeer[myid=3](plain=/0:0:0:0:0:0:0:0:13924)(secure=disabled):FileSnap@83] - Reading snapshot /home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/test/tmp/test1465497725988500626.junit.dir/data/version-2/snapshot.100000001
2016-09-13 05:09:32,271 [myid:2] - INFO [QuorumPeer[myid=2](plain=/127.0.0.1:13921)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id2,name1=replica.2,name2=Leader]
2016-09-13 05:09:32,272 [myid:2] - WARN [QuorumPeer[myid=2](plain=/127.0.0.1:13921)(secure=disabled):QuorumPeer@1127] - Unexpected exception
java.lang.InterruptedException
at java.lang.Object.wait(Native Method)
at org.apache.zookeeper.server.quorum.Leader.lead(Leader.java:561)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1124)
2016-09-13 05:09:32,272 [myid:2] - INFO [QuorumPeer[myid=2](plain=/127.0.0.1:13921)(secure=disabled):Leader@623] - Shutting down
2016-09-13 05:09:32,272 [myid:2] - WARN [QuorumPeer[myid=2](plain=/127.0.0.1:13921)(secure=disabled):QuorumPeer@1158] - PeerState set to LOOKING
2016-09-13 05:09:32,272 [myid:2] - WARN [QuorumPeer[myid=2](plain=/127.0.0.1:13921)(secure=disabled):QuorumPeer@1140] - QuorumPeer main thread exited
2016-09-13 05:09:32,272 [myid:2] - INFO [QuorumPeer[myid=2](plain=/127.0.0.1:13921)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id2]
2016-09-13 05:09:32,272 [myid:2] - INFO [QuorumPeer[myid=2](plain=/127.0.0.1:13921)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id2,name1=replica.2]
2016-09-13 05:09:32,272 [myid:2] - INFO [QuorumPeer[myid=2](plain=/127.0.0.1:13921)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id2,name1=replica.0]
2016-09-13 05:09:32,273 [myid:2] - INFO [QuorumPeer[myid=2](plain=/127.0.0.1:13921)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id2,name1=replica.1]
2016-09-13 05:09:32,273 [myid:2] - INFO [QuorumPeer[myid=2](plain=/127.0.0.1:13921)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id2,name1=replica.3]
2016-09-13 05:09:32,274 [myid:2] - INFO [/127.0.0.1:13923:QuorumCnxManager$Listener@661] - Leaving listener
2016-09-13 05:09:32,274 [myid:] - INFO [main:QuorumBase@398] - Shutting down leader election QuorumPeer[myid=2](plain=/127.0.0.1:13921)(secure=disabled)
2016-09-13 05:09:32,274 [myid:] - INFO [main:QuorumBase@403] - Waiting for QuorumPeer[myid=2](plain=/127.0.0.1:13921)(secure=disabled) to exit thread
2016-09-13 05:09:32,274 [myid:] - INFO [main:QuorumBase@394] - Shutting down quorum peer QuorumPeer[myid=3](plain=/0:0:0:0:0:0:0:0:13924)(secure=disabled)
2016-09-13 05:09:32,275 [myid:3] - INFO [ConnnectionExpirer:NIOServerCnxnFactory$ConnectionExpirerThread@583] - ConnnectionExpirerThread interrupted
2016-09-13 05:09:32,275 [myid:3] - INFO [NIOServerCxnFactory.AcceptThread:0.0.0.0/0.0.0.0:13924:NIOServerCnxnFactory$AcceptThread@219] - accept thread exitted run method
2016-09-13 05:09:32,276 [myid:3] - INFO [NIOServerCxnFactory.SelectorThread-0:NIOServerCnxnFactory$SelectorThread@420] - selector thread exitted run method
2016-09-13 05:09:32,276 [myid:3] - INFO [QuorumPeer[myid=3](plain=/0:0:0:0:0:0:0:0:13924)(secure=disabled):FastLeaderElection@894] - New election. My id = 3, proposed zxid=0x100000004
2016-09-13 05:09:32,277 [myid:3] - WARN [WorkerSender[myid=3]:QuorumCnxManager@455] - Cannot open channel to 0 at election address /127.0.0.1:13917
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:441)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:482)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:419)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:486)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:465)
at java.lang.Thread.run(Thread.java:745)
2016-09-13 05:09:32,277 [myid:3] - INFO [/127.0.0.1:13926:QuorumCnxManager$Listener@661] - Leaving listener
2016-09-13 05:09:32,277 [myid:3] - WARN [WorkerSender[myid=3]:QuorumCnxManager@455] - Cannot open channel to 0 at election address /127.0.0.1:13917
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:441)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:489)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:419)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:486)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:465)
at java.lang.Thread.run(Thread.java:745)
2016-09-13 05:09:32,278 [myid:3] - WARN [WorkerSender[myid=3]:QuorumCnxManager@455] - Cannot open channel to 1 at election address /127.0.0.1:13920
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:441)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:482)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:419)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:486)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:465)
at java.lang.Thread.run(Thread.java:745)
2016-09-13 05:09:32,278 [myid:] - INFO [main:QuorumBase@398] - Shutting down leader election QuorumPeer[myid=3](plain=/0:0:0:0:0:0:0:0:13924)(secure=disabled)
2016-09-13 05:09:32,279 [myid:] - INFO [main:QuorumBase@403] - Waiting for QuorumPeer[myid=3](plain=/0:0:0:0:0:0:0:0:13924)(secure=disabled) to exit thread
2016-09-13 05:09:32,279 [myid:3] - WARN [WorkerSender[myid=3]:QuorumCnxManager@455] - Cannot open channel to 1 at election address /127.0.0.1:13920
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:441)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:489)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:419)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:486)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:465)
at java.lang.Thread.run(Thread.java:745)
2016-09-13 05:09:32,280 [myid:3] - INFO [WorkerSender[myid=3]:FastLeaderElection$Messenger$WorkerSender@470] - WorkerSender is down
2016-09-13 05:09:32,280 [myid:3] - INFO [QuorumPeer[myid=3](plain=/0:0:0:0:0:0:0:0:13924)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id3,name1=replica.3,name2=LeaderElection]
2016-09-13 05:09:32,280 [myid:3] - WARN [QuorumPeer[myid=3](plain=/0:0:0:0:0:0:0:0:13924)(secure=disabled):QuorumPeer@1140] - QuorumPeer main thread exited
2016-09-13 05:09:32,280 [myid:3] - INFO [QuorumPeer[myid=3](plain=/0:0:0:0:0:0:0:0:13924)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id3]
2016-09-13 05:09:32,280 [myid:3] - INFO [QuorumPeer[myid=3](plain=/0:0:0:0:0:0:0:0:13924)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id3,name1=replica.3]
2016-09-13 05:09:32,281 [myid:3] - INFO [QuorumPeer[myid=3](plain=/0:0:0:0:0:0:0:0:13924)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id3,name1=replica.0]
2016-09-13 05:09:32,281 [myid:3] - INFO [QuorumPeer[myid=3](plain=/0:0:0:0:0:0:0:0:13924)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id3,name1=replica.1]
2016-09-13 05:09:32,281 [myid:3] - INFO [QuorumPeer[myid=3](plain=/0:0:0:0:0:0:0:0:13924)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id3,name1=replica.2]
2016-09-13 05:09:32,283 [myid:] - INFO [main:ZKTestCase$1@70] - FAILED testDuringLeaderSync
java.lang.AssertionError: zoo.cfg.dynamic.next is not deleted.
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.junit.Assert.assertFalse(Assert.java:64)
at org.apache.zookeeper.server.quorum.ReconfigDuringLeaderSyncTest.testDuringLeaderSync(ReconfigDuringLeaderSyncTest.java:155)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:79)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.TestWatchman$1.evaluate(TestWatchman.java:53)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:38)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:535)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1182)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:1033)
2016-09-13 05:09:32,284 [myid:] - INFO [main:ZKTestCase$1@60] - FINISHED testDuringLeaderSync
{noformat}
flaky, flaky-test 9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
2 years, 34 weeks ago 0|i33l1r:
ZooKeeper ZOOKEEPER-2576

After svn to git migration ZooKeeper Precommit jenkins job is failing.

Bug Resolved Blocker Fixed Patrick D. Hunt Patrick D. Hunt Patrick D. Hunt 12/Sep/16 12:46   16/Oct/16 10:58 12/Sep/16 23:23   3.6.0 build   0 3   After moving from svn to git the precommit job is failing. I've disabled it temporarily.
https://builds.apache.org/view/S-Z/view/ZooKeeper/job/PreCommit-ZOOKEEPER-Build/
9223372036854775807 No Perforce job exists for this issue. 2 9223372036854775807
3 years, 27 weeks, 2 days ago 0|i33j1z:
ZooKeeper ZOOKEEPER-2575

/./// does not have the form scheme:id:perm and client is quit.

Bug Open Minor Unresolved kevin.chen Prabhunath Yadav Prabhunath Yadav 12/Sep/16 01:34   23/Nov/16 04:41   3.3.3   java client   0 4   while creating node using command (random arguments like this).
create / /./// or some wrong format it shows the message
/./// does not have the form scheme:id:perm
with Exception in thread "main" org.apache.zookeeper.KeeperException$InvalidACLException: KeeperErrorCode=InvalidACL
.....

It should give the accurate message but it should not get closed or quit.
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 17 weeks, 1 day ago 0|i33i6n:
ZooKeeper ZOOKEEPER-2574

PurgeTxnLog can inadvertently delete required txn log files

Bug Closed Major Fixed Abhishek Rai Abhishek Rai Abhishek Rai 09/Sep/16 16:20   15/Nov/17 16:29 22/Jan/17 20:31 3.4.7, 3.4.8, 3.5.0, 3.5.1, 3.5.2 3.4.10, 3.5.3, 3.6.0 server   0 9   Zookeeper 3.4.8, standalone, and 3-server quorum As part of the fix for ZOOKEEPER-1797, the call to FileTxnSnapLog.getSnapshotLogs() was removed from PurgeTxnLog.java. As a result, some old-looking but required txn log files can be deleted, resulting in data corruption or loss.

For example, consider the following:

1. Configuration:
autopurge.snapRetainCount=3

2. Following files exist:
log.100 spans transactions from zxid=100 till zxid=140 (inclusive)
snapshot.110 - snapshot as of zxid=110
snapshot.120 - snapshot as of zxid=120
snapshot.130 - snapshot as of zxid=130

Above scenario is possible when snapshotting has happened multiple times but without accompanying log rollover, which is possible if the server was running as a learner.

3. PurgeTxnLog retains all snapshots but deletes log.100 because its zxid is older than the zxid of the oldest snapshot (110). This results in loss of transactions in the range 131-140.

Before the fix for ZOOKEEPER-1797, this was avoided by the call to FileTxnSnapLog.getSnapshotLogs() which finds and retains the newest txn log file with starting zxid < oldest retained snapshot's highest zxid.
9223372036854775807 No Perforce job exists for this issue. 6 9223372036854775807
2 years, 18 weeks, 1 day ago 0|i33glz:
ZooKeeper ZOOKEEPER-2573

Modify Info.REVISION to adapt git repo

Bug Closed Major Fixed Edward Ribeiro Mohammad Arshad Mohammad Arshad 09/Sep/16 09:35   11/Apr/17 18:45 25/Jan/17 07:52 3.4.9, 3.5.2 3.4.10, 3.5.3, 3.6.0 build, server   0 7   Modify {{org.apache.zookeeper.version.Info.REVISION}} to store git repo revision
Currently {{org.apache.zookeeper.version.Info.REVISION}} stores the svn repo revision which is of type int
But after migrating to git repo the git repo's revision(commit 63f5132716c08b3d8f18993bf98eb46eb42f80fb) can not be stored in this variable.
So either we should modify this variable to string to introduce new variable to store the git revision and leave the svn revision variable unchanged.
build.xml, and org.apache.zookeeper.version.util.VerGen also need to be modified.
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
2 years, 49 weeks, 2 days ago After the migration of ZooKeeper's version control system from 'svn repo' to 'apache git repo' the revision info becomes git's SHA-1 hash value. 0|i33fyn:
ZooKeeper ZOOKEEPER-2572

Potential resource leak in FileTxnLog.truncate

Bug Open Major Unresolved maoling Michael Han Michael Han 09/Sep/16 01:50   19/Feb/19 06:46   3.4.9, 3.5.2, 3.4.11   server   0 5   In FileTxnLog.truncate, we have:
{code}
public boolean truncate(long zxid) throws IOException {
FileTxnIterator itr = null;
try {
itr = new FileTxnIterator(this.logDir, zxid);
PositionInputStream input = itr.inputStream;
if(input == null) {
throw new IOException("No log files found to truncate! This could " +
"happen if you still have snapshots from an old setup or " +
"log files were deleted accidentally or dataLogDir was changed in zoo.cfg.");
}
long pos = input.getPosition();
// now, truncate at the current position
RandomAccessFile raf=new RandomAccessFile(itr.logFile,"rw");
raf.setLength(pos);
raf.close();
while(itr.goToNextLog()) {
if (!itr.logFile.delete()) {
LOG.warn("Unable to truncate {}", itr.logFile);
}
}
} finally {
close(itr);
}
return true;
}
{code}

{{raf}} here can be potentially in a state of not closed after leaving the method, if there is an (IO) exception thrown from setLength.
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
2 years, 28 weeks ago 0|i33fdj:
ZooKeeper ZOOKEEPER-2571

Potential resource leak in QuorumPeer.writeLongToFile

Bug Open Major Unresolved Unassigned Michael Han Michael Han 09/Sep/16 01:33   22/Jun/18 00:49   3.4.9, 3.4.11   server   0 1   In QuorumPeer.writeLongToFile we have:
{code}
try {
bw.write(Long.toString(value));
bw.flush();

out.flush();
} catch (IOException e) {
LOG.error("Failed to write new file " + file, e);
// worst case here the tmp file/resources(fd) are not cleaned up
// and the caller will be notified (IOException)
aborted = true;
out.abort();
throw e;
} finally {
if (!aborted) {
// if the close operation (rename) fails we'll get notified.
// worst case the tmp file may still exist
out.close();
}
}
{code}

So if any unchecked exception thrown during write (e.g. out of memory, you never know), the output stream will not be closed. The fix is can be made by having the flag set at the end of the try block instead of of in the catch block, which only catch a specific type of exception (which is what ZOOKEEPER-1835 did, thus the same issue does not exist in 3.5.x branch.).
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 27 weeks, 6 days ago 0|i33fbr:
ZooKeeper ZOOKEEPER-2570

ZooKeeper clients are timed out when ZooKeeper servers are very busy

Bug Open Critical Unresolved Mohammad Arshad Mohammad Arshad Mohammad Arshad 08/Sep/16 12:27   13/Oct/16 14:59           0 3   ZooKeeper clients are timed out when ZooKeeper servers are very busy. Clients throw below exception and fail all the pending operations
{code}
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss
at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
{code}
Clients log bellow information
{noformat}
2016-09-22 01:49:08,001 [myid:127.0.0.1:11228] - WARN [main-SendThread(127.0.0.1:11228):ClientCnxn$SendThread@1181] - Client session timed out, have not heard from server in 13908ms for sessionid 0x20000d21b280000
2016-09-22 01:49:08,001 [myid:127.0.0.1:11228] - INFO [main-SendThread(127.0.0.1:11228):ClientCnxn$SendThread@1229] - Client session timed out, have not heard from server in 13908ms for sessionid 0x20000d21b280000, closing socket connection and attempting reconnect
{noformat}
*STEPS TO REPRODECE:*
# Create multi operation
{code}
List<Op> ops = new ArrayList<Op>();
for (int i = 0; i < N; i++) {
Op create = Op.create(rootNode + "/" + i, "".getBytes(), ZooDefs.Ids.OPEN_ACL_UNSAFE,
CreateMode.PERSISTENT);
ops.add(create);
}
{code}
Chose N in such a way that the total multi operation request size is less than but near 1 MB. For bigger request size increase jute.maxbuffer in servers
# Submit the multi operation request
{code} zooKeeper.multi(ops);{code}
# After repeating above steps few times issue is reproduced
9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
3 years, 23 weeks ago 0|i33ei7:
ZooKeeper ZOOKEEPER-2569

plain password is stored when set individual ACL using digest scheme

Bug Open Major Unresolved Rakesh Kumar Singh Rakesh Kumar Singh Rakesh Kumar Singh 08/Sep/16 09:58   23/Sep/16 01:05   3.5.1   security   0 3   Plain password is stored when set individual ACL using digest scheme instead of storing the username and encoded hash string of <username:password>

[zk: localhost:2181(CONNECTED) 13] addauth digest user:pass
[zk: localhost:2181(CONNECTED) 14] setAcl /newNode digest:user:pass:crdwa
[zk: localhost:2181(CONNECTED) 15] getAcl /newNode
'digest,'user:pass
: cdrwa
[zk: localhost:2181(CONNECTED) 16]
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 26 weeks ago 0|i33ean:
ZooKeeper ZOOKEEPER-2568

Node created having name with space is not deleted with delete command

Bug Resolved Minor Information Provided Unassigned Prabhunath Yadav Prabhunath Yadav 08/Sep/16 07:18   12/Sep/16 22:19 08/Sep/16 18:36 3.4.6   java client, server   0 1   In java using For Example :
String myNode="/MyNode"+new Date() ;
connector.createNode(newNode, new Date().toString().getBytes());

and createNode is defined as:
public void createNode(String path, byte[] data) throws Exception
{
zk.create(path, data, Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT);
}

if we delete the node from command
delete /MyNodeFri Aug 12 09:42:16 GMT+05:30 2016
then we get exception saying
Command failed:java.lang.NumberFormatException: for input string : "Aug"

How to delete such node ? may rmr command can remove but why delete command not working ?
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
Important
3 years, 27 weeks, 2 days ago 0|i33e1z:
ZooKeeper ZOOKEEPER-2567

Error message is not correct when wrong argument is passed for "reconfig" cmd

Bug Open Minor Unresolved Rakesh Kumar Singh Rakesh Kumar Singh Rakesh Kumar Singh 08/Sep/16 07:00   23/Nov/16 09:53       java client   0 3   Error message is not correct when wrong argument is passed for "reconfig" cmd

Steps to reproduce:-
1. Start zookeeper in cluster mode
2. use reconfig cmd with wrong argument (pass : instead of ;)
[zk: localhost:2181(CONNECTED) 10] reconfig -remove 3 -add 3=10.18.221.194:2888:3888:2181
KeeperErrorCode = BadArguments for

Here error message is not complete and informative on client console.

The log is as below:-
2016-09-08 18:54:08,701 [myid:1] - INFO [ProcessThread(sid:1 cport:-1)::PrepRequestProcessor@512] - Incremental reconfig
2016-09-08 18:54:08,702 [myid:1] - INFO [ProcessThread(sid:1 cport:-1)::PrepRequestProcessor@843] - Got user-level KeeperException when processing sessionid:0x100299b7eac0000 type:reconfig cxid:0x7 zxid:0x400000004 txntype:-1 reqpath:n/a Error Path:Reconfiguration failed Error:KeeperErrorCode = BadArguments for Reconfiguration failed
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 26 weeks, 2 days ago 0|i33e0v:
ZooKeeper ZOOKEEPER-2566

space should be truncated while reading password for keystore/truststore which is required to configure while SSL enabled

Bug Open Minor Unresolved Unassigned Athyab Ameer Athyab Ameer 08/Sep/16 05:51   08/Sep/16 18:37   3.5.1   server   0 2   space should be truncated while reading password for keystore/truststore which is required to configure while SSL enabled.
As of now if we configure the password with any heading/trailing space, the zookeeper server will fail to start.
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 28 weeks ago 0|i33dxb:
ZooKeeper ZOOKEEPER-2565

listquota <path> should display the quota even it is set on parent/child node

Bug Open Minor Unresolved kevin.chen Rakesh Kumar Singh Rakesh Kumar Singh 08/Sep/16 05:29   26/May/19 02:47   3.5.1   server   0 7 0 2400   listquota <path> should display the quota even it is set on parent/child node. As of now if we have a parent-child hierarchy for example n1->n2->n3 and quota is set for n2. If we try to get quota details on n1 and n3 using listquota, it says no quota set but if try to set the quota on those nodes it fails saying quota is already set on parent node...
So listquota should fetch the quota set on any node in hierarchy with exact path (on which quota is set) even though this api is called on any other node in that hierarchy.
100% 100% 2400 0 pull-request-available 9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 11 weeks, 2 days ago 0|i33duf:
ZooKeeper ZOOKEEPER-2564

No message is prompted when trying to delete quota with different quota option

Bug Open Minor Unresolved Rakesh Kumar Singh Rakesh Kumar Singh Rakesh Kumar Singh 08/Sep/16 02:57   28/May/19 06:09   3.5.1   server   0 4   No message is prompted when trying to delete quota with different quota option.

Steps to reproduce:-
1. Start zookeeper in cluster mode
2. Create some node and set quota like
setquota -n 10 /test
3. Now try to delete as below:-
delquota -b /test

Here no message/exception is prompted. We should prompt message like
"Byte Quota does not exist for /test"
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
42 weeks, 2 days ago 0|i33dlb:
ZooKeeper ZOOKEEPER-2563

A revisit to setquota

Bug Resolved Major Fixed Rakesh Kumar Singh Rakesh Kumar Singh Rakesh Kumar Singh 08/Sep/16 02:19   17/Jul/19 17:33 17/Jul/19 07:42 3.5.1 3.6.0 server   0 3 0 7200   delquota -[n|b] is not deleting the set quota properly

Steps to reproduce:-
1. Start zookeeper in cluster mode (ssl)
2. create some node say /test
3. Run command as listquota says (as expected)
quota for /test does not exist
4. setquota let say
setquota -n 10 /test
5. Now try to delete this as below
delquota -n /test
6. now check the quota

[zk: localhost:2181(CONNECTED) 1] listquota /test
absolute path is /zookeeper/quota/test/zookeeper_limits
Output quota for /test count=-1,bytes=-1
Output stat for /test count=1,bytes=5

7. Here it is not deleted quota node for test
8. Now try to set some new quota
It fails as it is not deleted correctly while delete

[zk: localhost:2181(CONNECTED) 3] setquota -n 11 /test
Command failed: java.lang.IllegalArgumentException: /test has a parent /zookeeper/quota/test which has a quota

But through delquota it is able to delete
100% 100% 7200 0 pull-request-available 9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
35 weeks, 1 day ago 0|i33dj3:
ZooKeeper ZOOKEEPER-2562

Safely persist renames of *epoch.tmp to *epoch by issuing fsync on parent directory -- Possible cluster unavailability otherwise

Bug Open Major Unresolved Unassigned Ramnatthan Alagappan Ramnatthan Alagappan 07/Sep/16 14:54   07/Sep/16 14:54     3.4.7     0 1   Three node linux cluster I am running a three node ZooKeeper cluster.

Renames of acceptedEpoch.tmp to acceptedEpoch and currentEpoch.tmp to currentEpoch have to persisted to disk by explicitly issuing fsync on the parent directory. If not, the rename might not hit the disk immediately and if a crash occurs at this point, then the server would fail to start with the following error in the log. If this happens on two more or nodes, then the cluster can become unavailable.

[myid:] - INFO [main:QuorumPeerConfig@103] - Reading configuration from: /tmp/zoo2.cfg
[myid:] - INFO [main:QuorumPeer$QuorumServer@149] - Resolved hostname: 127.0.0.2 to address: /127.0.0.2
[myid:] - INFO [main:QuorumPeer$QuorumServer@149] - Resolved hostname: 127.0.0.4 to address: /127.0.0.4
[myid:] - INFO [main:QuorumPeer$QuorumServer@149] - Resolved hostname: 127.0.0.3 to address: /127.0.0.3
[myid:] - INFO [main:QuorumPeerConfig@331] - Defaulting to majority quorums
[myid:1] - INFO [main:DatadirCleanupManager@78] - autopurge.snapRetainCount set to 3
[myid:1] - INFO [main:DatadirCleanupManager@79] - autopurge.purgeInterval set to 0
[myid:1] - INFO [main:DatadirCleanupManager@101] - Purge task is not scheduled.
[myid:1] - INFO [main:QuorumPeerMain@127] - Starting quorum peer
[myid:1] - INFO [main:NIOServerCnxnFactory@89] - binding to port 0.0.0.0/0.0.0.0:2182
[myid:1] - INFO [main:QuorumPeer@1019] - tickTime set to 2000
[myid:1] - INFO [main:QuorumPeer@1039] - minSessionTimeout set to -1
[myid:1] - INFO [main:QuorumPeer@1050] - maxSessionTimeout set to -1
[myid:1] - INFO [main:QuorumPeer@1065] - initLimit set to 5
[myid:1] - INFO [main:FileSnap@83] - Reading snapshot /run/shm/dice-4636/113-98-129-z_majority_RO_OM_0=60_1=55/rdir-0/version-2/snapshot.100000002
[myid:1] - ERROR [main:QuorumPeer@557] - Unable to load database on disk
java.io.IOException: The accepted epoch, 1 is less than the current epoch, 2
at org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase(QuorumPeer.java:554)
at org.apache.zookeeper.server.quorum.QuorumPeer.start(QuorumPeer.java:500)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:153)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:111)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:78)
2016-04-15 03:24:57,144 [myid:1] - ERROR [main:QuorumPeerMain@89] - Unexpected exception, exiting abnormally
java.lang.RuntimeException: Unable to run quorum server
at org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase(QuorumPeer.java:558)
at org.apache.zookeeper.server.quorum.QuorumPeer.start(QuorumPeer.java:500)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:153)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:111)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:78)
Caused by: java.io.IOException: The accepted epoch, 1 is less than the current epoch, 2
at org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase(QuorumPeer.java:554)
... 4 more

Similarly, when new log file is created, the parent directory needs be explicitly fsynced to persist the log file. Otherwise a data loss might be possible (We have reproduced the above issues). Please see this: https://www.quora.com/Linux/When-should-you-fsync-the-containing-directory-in-addition-to-the-file-itself and http://research.cs.wisc.edu/wind/Publications/alice-osdi14.pdf.
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 28 weeks, 1 day ago 0|i33cvb:
ZooKeeper ZOOKEEPER-2561

CLONE - Possible Cluster Unavailability

Bug Open Major Unresolved Unassigned Athyab Ameer Athyab Ameer 07/Sep/16 13:40   07/Sep/16 13:40     3.4.8 server   0 1   Three node linux cluster Possible Cluster Unvailability

I am running a three node ZooKeeper cluster. Each node runs Linux.

I see the below sequence of system calls when ZooKeeper appends a user data item to the log file.

1 write("/data/version-2/log.200000001", offset=65, count=12)
2 write("/data/version-2/log.200000001", offset=77, count=16323)
3 write("/data/version-2/log.200000001", offset=16400, count=4209)
4 write("/data/version-2/log.200000001", offset=20609, count=1)
5 fdatasync("/data//version-2/log.200000001")

Now, a crash could happen just after operation 4 but before the final fdatasync. In this situation, the file system could persist the 4th operation and fail to persist the 3rd operation because of the crash and there is fsync in between them. In such cases, ZooKeeper server fails to start with the following messages in its log file:

[myid:] - INFO [main:QuorumPeerConfig@103] - Reading configuration from: /tmp/zoo2.cfg
[myid:] - INFO [main:QuorumPeer$QuorumServer@149] - Resolved hostname: 127.0.0.2 to address: /127.0.0.2
[myid:] - INFO [main:QuorumPeer$QuorumServer@149] - Resolved hostname: 127.0.0.4 to address: /127.0.0.4
[myid:] - INFO [main:QuorumPeer$QuorumServer@149] - Resolved hostname: 127.0.0.3 to address: /127.0.0.3
[myid:] - INFO [main:QuorumPeerConfig@331] - Defaulting to majority quorums
[myid:1] - INFO [main:DatadirCleanupManager@78] - autopurge.snapRetainCount set to 3
[myid:1] - INFO [main:DatadirCleanupManager@79] - autopurge.purgeInterval set to 0
[myid:1] - INFO [main:DatadirCleanupManager@101] - Purge task is not scheduled.
[myid:1] - INFO [main:QuorumPeerMain@127] - Starting quorum peer
[myid:1] - INFO [main:NIOServerCnxnFactory@89] - binding to port 0.0.0.0/0.0.0.0:2182
[myid:1] - INFO [main:QuorumPeer@1019] - tickTime set to 2000
[myid:1] - INFO [main:QuorumPeer@1039] - minSessionTimeout set to -1
[myid:1] - INFO [main:QuorumPeer@1050] - maxSessionTimeout set to -1
[myid:1] - INFO [main:QuorumPeer@1065] - initLimit set to 5
[myid:1] - INFO [main:FileSnap@83] - Reading snapshot /data/version-2/snapshot.100000002
[myid:1] - ERROR [main:QuorumPeer@557] - Unable to load database on disk
java.io.IOException: CRC check failed
at org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.next(FileTxnLog.java:635)
at org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:158)
at org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:223)
at org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase(QuorumPeer.java:510)
at org.apache.zookeeper.server.quorum.QuorumPeer.start(QuorumPeer.java:500)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:153)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:111)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:78)
2016-04-15 04:00:32,795 [myid:1] - ERROR [main:QuorumPeerMain@89] - Unexpected exception, exiting abnormally
java.lang.RuntimeException: Unable to run quorum server
at org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase(QuorumPeer.java:558)
at org.apache.zookeeper.server.quorum.QuorumPeer.start(QuorumPeer.java:500)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:153)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:111)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:78)
Caused by: java.io.IOException: CRC check failed
at org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.next(FileTxnLog.java:635)
at org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:158)
at org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:223)
at org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase(QuorumPeer.java:510)
... 4 more

The same happens when the 3rd and 4th writes hit the disk but the 2nd operation does not.

Now, two nodes of a three node cluster can easily reach this state, rendering the entire cluster unavailable. ZooKeeper, on recovery should be able to handle such checksum mismatches gracefully to maintain cluster availability.
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 28 weeks, 1 day ago 0|i33cpz:
ZooKeeper ZOOKEEPER-2560

Possible Cluster Unavailability

Bug Open Major Unresolved Unassigned Ramnatthan Alagappan Ramnatthan Alagappan 07/Sep/16 11:12   07/Sep/16 13:40     3.4.8 server   0 2   Three node linux cluster Possible Cluster Unvailability

I am running a three node ZooKeeper cluster. Each node runs Linux.

I see the below sequence of system calls when ZooKeeper appends a user data item to the log file.

1 write("/data/version-2/log.200000001", offset=65, count=12)
2 write("/data/version-2/log.200000001", offset=77, count=16323)
3 write("/data/version-2/log.200000001", offset=16400, count=4209)
4 write("/data/version-2/log.200000001", offset=20609, count=1)
5 fdatasync("/data//version-2/log.200000001")

Now, a crash could happen just after operation 4 but before the final fdatasync. In this situation, the file system could persist the 4th operation and fail to persist the 3rd operation because of the crash and there is fsync in between them. In such cases, ZooKeeper server fails to start with the following messages in its log file:

[myid:] - INFO [main:QuorumPeerConfig@103] - Reading configuration from: /tmp/zoo2.cfg
[myid:] - INFO [main:QuorumPeer$QuorumServer@149] - Resolved hostname: 127.0.0.2 to address: /127.0.0.2
[myid:] - INFO [main:QuorumPeer$QuorumServer@149] - Resolved hostname: 127.0.0.4 to address: /127.0.0.4
[myid:] - INFO [main:QuorumPeer$QuorumServer@149] - Resolved hostname: 127.0.0.3 to address: /127.0.0.3
[myid:] - INFO [main:QuorumPeerConfig@331] - Defaulting to majority quorums
[myid:1] - INFO [main:DatadirCleanupManager@78] - autopurge.snapRetainCount set to 3
[myid:1] - INFO [main:DatadirCleanupManager@79] - autopurge.purgeInterval set to 0
[myid:1] - INFO [main:DatadirCleanupManager@101] - Purge task is not scheduled.
[myid:1] - INFO [main:QuorumPeerMain@127] - Starting quorum peer
[myid:1] - INFO [main:NIOServerCnxnFactory@89] - binding to port 0.0.0.0/0.0.0.0:2182
[myid:1] - INFO [main:QuorumPeer@1019] - tickTime set to 2000
[myid:1] - INFO [main:QuorumPeer@1039] - minSessionTimeout set to -1
[myid:1] - INFO [main:QuorumPeer@1050] - maxSessionTimeout set to -1
[myid:1] - INFO [main:QuorumPeer@1065] - initLimit set to 5
[myid:1] - INFO [main:FileSnap@83] - Reading snapshot /data/version-2/snapshot.100000002
[myid:1] - ERROR [main:QuorumPeer@557] - Unable to load database on disk
java.io.IOException: CRC check failed
at org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.next(FileTxnLog.java:635)
at org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:158)
at org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:223)
at org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase(QuorumPeer.java:510)
at org.apache.zookeeper.server.quorum.QuorumPeer.start(QuorumPeer.java:500)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:153)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:111)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:78)
2016-04-15 04:00:32,795 [myid:1] - ERROR [main:QuorumPeerMain@89] - Unexpected exception, exiting abnormally
java.lang.RuntimeException: Unable to run quorum server
at org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase(QuorumPeer.java:558)
at org.apache.zookeeper.server.quorum.QuorumPeer.start(QuorumPeer.java:500)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:153)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:111)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:78)
Caused by: java.io.IOException: CRC check failed
at org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.next(FileTxnLog.java:635)
at org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:158)
at org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:223)
at org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase(QuorumPeer.java:510)
... 4 more

The same happens when the 3rd and 4th writes hit the disk but the 2nd operation does not.

Now, two nodes of a three node cluster can easily reach this state, rendering the entire cluster unavailable. ZooKeeper, on recovery should be able to handle such checksum mismatches gracefully to maintain cluster availability.
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 28 weeks, 1 day ago 0|i33cdj:
ZooKeeper ZOOKEEPER-2559

the quota should be deleted when its parasitic path doesn't exist

Bug Open Major Unresolved Rakesh Kumar Singh Rakesh Kumar Singh Rakesh Kumar Singh 07/Sep/16 08:14   01/Jul/19 03:06   3.5.1, 3.5.2   server   0 1 0 1200   Failed to delete the set quota for ephemeral node when the node is deleted because of client session closed

[zk: localhost:2181(CONNECTED) 0] create -e /e_test hello
Created /e_test
[zk: localhost:2181(CONNECTED) 1] setquota -n 10 /e_test
[zk: localhost:2181(CONNECTED) 2] listquota /e_test
absolute path is /zookeeper/quota/e_test/zookeeper_limits
Output quota for /e_test count=10,bytes=-1
Output stat for /e_test count=1,bytes=5

Now close the client connection and so the ephemeral node gets deleted. But the corresponding quota is not getting deleted as below:-

[zk: localhost:2181(CONNECTED) 0] ls /
[test, test1, test3, zookeeper]
[zk: localhost:2181(CONNECTED) 1] listquota /e_test
absolute path is /zookeeper/quota/e_test/zookeeper_limits
Output quota for /e_test count=10,bytes=-1
Output stat for /e_test count=0,bytes=0
[zk: localhost:2181(CONNECTED) 2]


and so now again create the ephemeral node with same node and try to set the quota, it will fail.

[zk: localhost:2181(CONNECTED) 2] create -e /e_test hello
Created /e_test
[zk: localhost:2181(CONNECTED) 3] setquota -n 10 /e_test
Command failed: java.lang.IllegalArgumentException: /e_test has a parent /zookeeper/quota/e_test which has a quota
[zk: localhost:2181(CONNECTED) 4]
100% 100% 1200 0 pull-request-available 9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 28 weeks, 1 day ago 0|i33bzz:
ZooKeeper ZOOKEEPER-2558

Potential memory leak in recordio.c

Bug Closed Minor Fixed Michael Han Michael Han Michael Han 06/Sep/16 18:57   31/Mar/17 05:01 07/Sep/16 22:59 3.4.9, 3.5.2 3.4.10, 3.5.3, 3.6.0 c client   0 3   We have code like this in {{create_buffer_iarchive}} and {{create_buffer_oarchive}}:

{code}
struct iarchive *ia = malloc(sizeof(*ia));
struct buff_struct *buff = malloc(sizeof(struct buff_struct));
if (!ia) return 0;
if (!buff) {
free(ia);
return 0;
}
{code}

If first malloc failed but second succeeds, then the memory allocated with second malloc will not get freed when the function returned. One could argue that if first malloc failed the second will also fail (i.e. when system run out of memory), but I could also see the possibility of the opposite (the first malloc failed because heap fragmentation but the second succeeds).
9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
3 years, 28 weeks ago
Reviewed
0|i33b4n:
ZooKeeper ZOOKEEPER-2557

Update gitignore to account for other file extensions

Improvement Closed Trivial Fixed Edward Ribeiro Edward Ribeiro Edward Ribeiro 06/Sep/16 11:44   31/Mar/17 05:01 08/Sep/16 17:40 3.4.8 3.4.10, 3.5.3, 3.6.0     1 6   We are in the process of moving from subversion to git, but I have seen that the current ZK's {{gitignore}} doesn't account for many spurious types of files (e.g., *.swp, *.tmp) as well as other files created by IDEs (Eclipse, Intellij and NetBeans), among other file extensions. easyfix 9223372036854775807 No Perforce job exists for this issue. 2 9223372036854775807
3 years, 28 weeks ago
Reviewed
0|i33aen:
ZooKeeper ZOOKEEPER-2556

peerType remains as "observer" in zoo.cfg even though we change the node from observer to participant runtime

Bug Resolved Minor Fixed Rakesh Kumar Singh Rakesh Kumar Singh Rakesh Kumar Singh 06/Sep/16 06:43   03/Nov/16 12:48 03/Nov/16 11:41 3.5.1, 3.5.2   server   0 5   peerType remains as "observer" in zoo.cfg even though we change the node from observer to participant runtime

Steps to reproduce:-
1. Start zookeeper in cluster with one node as observer by configuring
peerType=observer in zoo.cfg and server.2=10.18.219.50:2888:3888:observer;2181
2. Start the cluster
3. start a client and change the node from observer to participant, the configuration related to peertype remained same though other things like clientport got from zoo.cfg

>reconfig -remove 2 -add 2=10.18.219.50:2888:3888:participant;2181

We should either remove this parameter or update with correct node type at run time
9223372036854775807 No Perforce job exists for this issue. 3 9223372036854775807
3 years, 20 weeks ago Committed. Thanks again Rakesh! 0|i339xz:
ZooKeeper ZOOKEEPER-2555

zookeeper started in observer mode takes long time to come to observer state once leader restarted and so client connected to observer mode has to wait for longer time to get service

Bug Open Major Unresolved Unassigned Rakesh Kumar Singh Rakesh Kumar Singh 06/Sep/16 05:59   06/Sep/16 06:00   3.5.1   server   0 1   zookeeper started in observer mode takes long time (some times 25 seconds) to come to observer state once leader restarted and so client connected to observer mode has to wait for longer time to get service

Steps to reproduce:-
1. Start zookeeper in cluster mode in which one node is in observer mode
2. stop the leader node (some times we need to wait for 30 secs to reproduce this issue)
3. Start the leader node
4. Check the observer node status -
It will be in
"Error contacting service. It is probably not running."

and takes long time (25 secs) to come to observer mode. And hence client connected to this node will not get service during this time.
Log at observer node is as below:-

2016-09-06 17:49:14,774 [myid:2] - WARN [WorkerSender[myid=2]:QuorumCnxManager@459] - Cannot open channel to 3 at election address /10.18.221.194:3888
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:391)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:444)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:485)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:421)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:486)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:465)
at java.lang.Thread.run(Thread.java:722)
{color:red}2016-09-06 17:49:14,776 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@688] - Notification: 2 (message format version), 3 (n.leader), 0x2a00000001 (n.zxid), 0x7 (n.round), LOOKING (n.state), 1 (n.sid), 0x2b (n.peerEPoch), LOOKING (my state)100000000 (n.config version)
2016-09-06 17:49:40,377 [myid:2] - INFO [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):FastLeaderElection@928] - Notification time out: 51200 {color}
2016-09-06 17:49:40,378 [myid:2] - INFO [WorkerSender[myid=2]:QuorumCnxManager@278] - Have smaller server identifier, so dropping the connection: (3, 2)
2016-09-06 17:49:40,379 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@688] - Notification: 2 (message format version), 3 (n.leader), 0x2a00000001 (n.zxid), 0x7 (n.round), FOLLOWING (n.state), 1 (n.sid), 0x2c (n.peerEPoch), LOOKING (my state)100000000 (n.config version)
2016-09-06 17:49:40,381 [myid:2] - INFO [/10.18.219.50:3888:QuorumCnxManager$Listener@637] - Received connection request /10.18.221.194:34085
2016-09-06 17:49:40,388 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@688] - Notification: 2 (message format version), 3 (n.leader), 0x2a00000001 (n.zxid), 0x7 (n.round), LEADING (n.state), 3 (n.sid), 0x2c (n.peerEPoch), LOOKING (my state)100000000 (n.config version)
2016-09-06 17:49:40,388 [myid:2] - INFO [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):MBeanRegistry@119] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id2,name1=replica.2,name2=LeaderElection]
2016-09-06 17:49:40,389 [myid:2] - INFO [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):QuorumPeer@1049] - OBSERVING
2016-09-06 17:49:40,389 [myid:2] - INFO [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):ZooKeeperServer@858] - minSessionTimeout set to 4000
2016-09-06 17:49:40,389 [myid:2] - INFO [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):ZooKeeperServer@867] - maxSessionTimeout set to 40000
2016-09-06 17:49:40,389 [myid:2] - INFO [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):ZooKeeperServer@156] - Created server with tickTime 2000 minSessionTimeout 4000 maxSessionTimeout 40000 datadir /home/Rakesh/Zookeeper/18_Aug/cluster/zookeeper-3.5.1-alpha/data/version-2 snapdir /home/Rakesh/Zookeeper/18_Aug/cluster/zookeeper-3.5.1-alpha/data/version-2
2016-09-06 17:49:40,389 [myid:2] - INFO [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):ObserverZooKeeperServer@56] - syncEnabled =true
2016-09-06 17:49:40,389 [myid:2] - INFO [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):Observer@72] - Observing /10.18.221.194:2888
2016-09-06 17:49:40,396 [myid:2] - INFO [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):FileSnap@83] - Reading snapshot /home/Rakesh/Zookeeper/18_Aug/cluster/zookeeper-3.5.1-alpha/data/version-2/snapshot.2a00000001
2016-09-06 17:49:40,410 [myid:2] - INFO [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):Learner@369] - Getting a snapshot from leader
2016-09-06 17:49:40,411 [myid:2] - INFO [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):Learner@509] - Learner received NEWLEADER message
2016-09-06 17:49:40,411 [myid:2] - INFO [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):FileTxnSnapLog@298] - Snapshotting: 0x2c00000000 to /home/Rakesh/Zookeeper/18_Aug/cluster/zookeeper-3.5.1-alpha/data/version-2/snapshot.2c00000000
2016-09-06 17:49:40,417 [myid:2] - INFO [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):Learner@493] - Learner received UPTODATE message
2016-09-06 17:49:40,417 [myid:2] - INFO [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):CommitProcessor@254] - Configuring CommitProcessor with 8 worker threads.
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 28 weeks, 2 days ago 0|i339wf:
ZooKeeper ZOOKEEPER-2554

reconfig can not add new server as observer

Bug Resolved Critical Not A Problem Mohammad Arshad Mohammad Arshad Mohammad Arshad 05/Sep/16 15:14   06/Sep/16 14:22 06/Sep/16 13:14 3.5.0, 3.5.1, 3.5.2   server   0 3   Try to add new observer server using reconfig API, server gets added as participant.
STEPS:
# create 3 node cluster.
{code}
server.0=127.0.0.1:11223:11224:participant;127.0.0.1:11222
server.1=127.0.0.1:11226:11227:participant;127.0.0.1:11225
server.2=127.0.0.1:11229:11230:participant;127.0.0.1:11228
{code}
# Suppose the 2 is the leader in the above cluster. Configure the new server as
{code}
server.2=127.0.0.1:11229:11230:participant;127.0.0.1:11228
server.3=127.0.0.1:11232:11233:observer;127.0.0.1:11231
{code}
# Connect to 1 and execute the reconfig command
{code}
zkClient.reconfig("server.3=127.0.0.1:11232:11233:observer;127.0.0.1:11231", null, null, -1, null, null);
{code}
# Verify sever 3. It was supposed to run as observer but it is running as participant
9223372036854775807 No Perforce job exists for this issue. 2 9223372036854775807
3 years, 28 weeks, 2 days ago 0|i339af:
ZooKeeper ZOOKEEPER-2553

ZooKeeper cluster unavailable due to corrupted log file during power failures -- java.io.IOException: Unreasonable length

Bug Open Major Unresolved Unassigned Ramnatthan Alagappan Ramnatthan Alagappan 05/Sep/16 13:20   15/Nov/19 08:38   3.4.8   server   1 4   Normal ZooKeeper cluster with 3 nodes running Linux I am running a three node ZooKeeper cluster.

When a new log file is created by ZooKeeper, I see the following sequence of system calls:

1. creat(new_log)
2. write(new_log, count=16) // This is a log header I believe/
3. truncate(new_log, from 16 bytes to 16 KBytes) // I have configured the log size to be 16K.

When the above sequence of operations complete, it is reasonable to expect the newly created log file to contain the header(16 bytes) and then filled with zeros till the end of the log.

But when a crash occurs (due to a power failure), while the truncate system call is in progress, it is possible for the log to contain garbage data when the system restarts from the crash. Note that if the crash occurs just after the truncate system call completes, then there is no problem. Basically, the truncate needs to be atomically persisted for ZooKeeper to recover from crashes correctly or (more realistically) the recovery code needs to deal with the case of expecting garbage in a newly created log.

As mentioned, if a crash occurs during the truncate system call, then ZooKeeper will fail to start with the following exception. Here is the stack trace:

java.io.IOException: Unreasonable length = -295704495
at org.apache.jute.BinaryInputArchive.checkLength(BinaryInputArchive.java:127)
at org.apache.jute.BinaryInputArchive.readBuffer(BinaryInputArchive.java:92)
at org.apache.zookeeper.server.persistence.Util.readTxnBytes(Util.java:233)
at org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.next(FileTxnLog.java:625)
at org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.next(FileTxnLog.java:652)
at org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.init(FileTxnLog.java:552)
at org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.<init>(FileTxnLog.java:527)
at org.apache.zookeeper.server.persistence.FileTxnLog.read(FileTxnLog.java:354)
at org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:132)
at org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:223)
at org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase(QuorumPeer.java:510)
at org.apache.zookeeper.server.quorum.QuorumPeer.start(QuorumPeer.java:500)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:153)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:111)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:78)
[myid:1] - ERROR [main:QuorumPeerMain@89] - Unexpected exception, exiting abnormally
java.lang.RuntimeException: Unable to run quorum server
at org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase(QuorumPeer.java:558)
at org.apache.zookeeper.server.quorum.QuorumPeer.start(QuorumPeer.java:500)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:153)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:111)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:78)
Caused by: java.io.IOException: Unreasonable length = -295704495
at org.apache.jute.BinaryInputArchive.checkLength(BinaryInputArchive.java:127)
at org.apache.jute.BinaryInputArchive.readBuffer(BinaryInputArchive.java:92)
at org.apache.zookeeper.server.persistence.Util.readTxnBytes(Util.java:233)
at org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.next(FileTxnLog.java:625)
at org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.next(FileTxnLog.java:652)
at org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.init(FileTxnLog.java:552)
at org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.<init>(FileTxnLog.java:527)
at org.apache.zookeeper.server.persistence.FileTxnLog.read(FileTxnLog.java:354)
at org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:132)
at org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:223)
at org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase(QuorumPeer.java:510)
... 4 more


Next, it is possible for two nodes of a 3-node ZooKeeper cluster to reach the same state. In that case, they both will fail to startup, rendering the entire cluster unavailable.
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
17 weeks, 6 days ago 0|i3397z:
ZooKeeper ZOOKEEPER-2552

Revisit release note doc and remove the items which are not related to the released version

Bug Closed Major Fixed Edward Ribeiro Rakesh Radhakrishnan Rakesh Radhakrishnan 05/Sep/16 08:43   12/May/17 21:05 15/Dec/16 05:14 3.4.9 3.4.10     0 4   Couple of issues listed on http://zookeeper.apache.org/
doc/r3.4.9/releasenotes.html that are either 'Open' or 'Patch available'. For example, issues were wrongly marked as "3.4.8" fix version in jira and has caused the trouble.

This jira to cross check all the jira issues present in the release note and check the correctness.
9223372036854775807 No Perforce job exists for this issue. 2 9223372036854775807
2 years, 44 weeks, 5 days ago 0|i338zb:
ZooKeeper ZOOKEEPER-2551

Remove Hadoop Logo from ZooKeeper documentation

Bug Patch Available Major Unresolved Mohammad Arshad Mohammad Arshad Mohammad Arshad 05/Sep/16 03:01   05/Feb/20 07:11     3.7.0, 3.5.8 documentation   0 1   ZooKeeper documentation has hadoop logo on each page's header. There is no significance to put the hadoop logo on ZooKeeper project. So hadoop logo should be removed from Zookeeper as ZooKeeper is independent of hadoop project, 9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
1 year, 45 weeks ago 0|i338nr:
ZooKeeper ZOOKEEPER-2550

FollowerResyncConcurrencyTest failed in ZooKeeper 3.3.3

Bug Open Blocker Unresolved Unassigned KangYin KangYin 05/Sep/16 02:05   08/Sep/16 01:36   3.3.3   leaderElection, quorum, server, tests   0 1   Windows 10,
Java 1.8.0,
IDEA 2016.1.4,
JUnit 4.8.1
I'm studying on the Test of ZooKeeper 3.3.3 but got a test failure when I run _testResyncBySnapThenDiffAfterFollowerCrashes_ in _FollowerResyncConcurrencyTest.java_.

{quote}
2016-09-05 13:57:35,072 - INFO [main:QuorumBase@307] - FINISHED testResyncBySnapThenDiffAfterFollowerCrashes

java.util.concurrent.TimeoutException: Did not connect

at org.apache.zookeeper.test.ClientBase$CountdownWatcher.waitForConnected(ClientBase.java:119)
at org.apache.zookeeper.test.FollowerResyncConcurrencyTest.testResyncBySnapThenDiffAfterFollowerCrashes(FollowerResyncConcurrencyTest.java:95)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at junit.framework.TestCase.runTest(TestCase.java:168)
at junit.framework.TestCase.runBare(TestCase.java:134)
at junit.framework.TestResult$1.protect(TestResult.java:110)
at junit.framework.TestResult.runProtected(TestResult.java:128)
at junit.framework.TestResult.run(TestResult.java:113)
at junit.framework.TestCase.run(TestCase.java:124)
at junit.framework.TestSuite.runTest(TestSuite.java:232)
at junit.framework.TestSuite.run(TestSuite.java:227)
at org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:83)
at org.junit.runner.JUnitCore.run(JUnitCore.java:157)
at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:119)
at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:42)
at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:234)
at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:74)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:144)
{quote}

Which happened in _FollowerResyncConcurrencyTest.java_ at line 92.

{quote}
index = (index == 1) ? 2 : 1;
qu.shutdown(index);
final ZooKeeper zk3 = new DisconnectableZooKeeper("127.0.0.1:" + qu.getPeer(3).peer.getClientPort(), 1000,watcher3);
{color:red}watcher3.waitForConnected(CONNECTION_TIMEOUT);{color}
zk3.create("/mybar", null, ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.EPHEMERAL_SEQUENTIAL);
{quote}

I checked the Log Message, and I guess it is probably because of the following ERROR (marked as blue):

{quote}
2016-09-05 13:56:54,928 - INFO [main-SendThread():ClientCnxn$SendThread@1041] - Opening socket connection to server /127.0.0.1:11237
2016-09-05 13:56:54,930 - INFO [main-SendThread(127.0.0.1:11237):ClientCnxn$SendThread@949] - Socket connection established to 127.0.0.1/127.0.0.1:11237, initiating session
2016-09-05 13:56:54,930 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11237:NIOServerCnxn$Factory@251] - Accepted socket connection from /127.0.0.1:33566
2016-09-05 13:56:54,957 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11237:NIOServerCnxn@777] - Client attempting to establish new session at /127.0.0.1:33566
{color:blue}
2016-09-05 13:56:55,000 - INFO [SyncThread:3:FileTxnLog@197] - Creating new log file: log.100000001
2016-09-05 13:56:55,000 - WARN [QuorumPeer:/0:0:0:0:0:0:0:0:11235:Follower@116] - Got zxid 0x100000001 expected 0x1
2016-09-05 13:56:55,000 - INFO [SyncThread:2:FileTxnLog@197] - Creating new log file: log.100000001
2016-09-05 13:56:55,078 - ERROR [CommitProcessor:3:CommitProcessor@146] - Unexpected exception causing CommitProcessor to exit
java.lang.AssertionError
at org.apache.zookeeper.jmx.MBeanRegistry.register(MBeanRegistry.java:66)
at org.apache.zookeeper.server.NIOServerCnxn.finishSessionInit(NIOServerCnxn.java:1552)
at org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:183)
at org.apache.zookeeper.server.quorum.Leader$ToBeAppliedRequestProcessor.processRequest(Leader.java:540)
at org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:73)
2016-09-05 13:56:55,078 - INFO [CommitProcessor:3:CommitProcessor@148] - CommitProcessor exited loop!
{color}

2016-09-05 13:56:55,931 - INFO [main-SendThread(127.0.0.1:11237):ClientCnxn$SendThread@1157] - Client session timed out, have not heard from server in 1001ms for sessionid 0x0, closing socket connection and attempting reconnect
2016-09-05 13:56:58,035 - INFO [main-SendThread(127.0.0.1:11237):ClientCnxn$SendThread@1041] - Opening socket connection to server 127.0.0.1/127.0.0.1:11237
2016-09-05 13:56:58,036 - INFO [main-SendThread(127.0.0.1:11237):ClientCnxn$SendThread@949] - Socket connection established to 127.0.0.1/127.0.0.1:11237, initiating session
{quote}

I'll very appreciate it if I can get some help from you genius people.
Thanks.
test 9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 28 weeks ago 0|i338mf:
ZooKeeper ZOOKEEPER-2549

As NettyServerCnxn.sendResponse() allows all the exception to bubble up it can stop main ZK requests processing thread

Bug Patch Available Major Unresolved Yuliya Feldman Yuliya Feldman Yuliya Feldman 03/Sep/16 02:39   25/Dec/19 21:26   3.5.1 3.7.0 server   0 5 0 600   As NettyServerCnxn.sendResponse() allows all the exception to bubble up it can stop main ZK requests processing thread and make Zookeeper server look like it is hanging, while it just can not process any request anymore.

Idea is to catch all the exceptions in NettyServerCnxn.sendResponse() , convert them to IOException and allow it propagating up
100% 100% 600 0 pull-request-available 9223372036854775807 No Perforce job exists for this issue. 8 9223372036854775807
12 weeks ago 0|i337rj:
ZooKeeper ZOOKEEPER-2548

zooInspector does not start on Windows

Bug Closed Major Fixed Mohammad Arshad Mohammad Arshad Mohammad Arshad 02/Sep/16 16:23   17/May/17 23:44 07/Sep/16 18:14   3.5.3, 3.6.0 contrib   0 3   ZooInspector is very usefully tool but seems its windows scripts are not maintained.
zooInspector.cmd commands fails with bellow error:
{noformat}
D:\workspace\ZooInspector>zooInspector.cmd

D:\workspace\ZooInspector>#!/bin/sh
'#!' is not recognized as an internal or external command,
operable program or batch file.

D:\workspace\ZooInspector># Licensed to the Apache Software Foundation (ASF) under one or more
'#' is not recognized as an internal or external command,
operable program or batch file.
{noformat}
9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
3 years, 28 weeks, 1 day ago
Reviewed
0|i337ef:
ZooKeeper ZOOKEEPER-2547

IP ACL is not working with NettyServerCnxnFactory

Bug Patch Available Major Unresolved Mohammad Arshad Mohammad Arshad Mohammad Arshad 02/Sep/16 13:16   05/Feb/20 07:11   3.5.0 3.7.0, 3.5.8     1 3   IP based ACL is not working with NettyServerCnxnFactory.

Scenario:
1) Configure serverCnxnFactory= org.apache.zookeeper.server.NettyServerCnxnFactory and start ZooKeeper server
2) Create a znode "/n" with ACL(ZooDefs.Perms.ALL, new Id("ip", "127.0.0.1/8")
3) Create child node /n/n1. Child node creation fails.
But the same above scenario works with NIOServerCnxnFactory
9223372036854775807 No Perforce job exists for this issue. 2 9223372036854775807
3 years, 26 weeks, 5 days ago 0|i3372v:
ZooKeeper ZOOKEEPER-2546

Started throwing "Error Path:null Error:KeeperErrorCode = ReconfigInProgress" error when trying to change the cluster using reconfig and the IO hangged at one node

Bug Open Major Unresolved Unassigned Rakesh Kumar Singh Rakesh Kumar Singh 02/Sep/16 07:56   02/Sep/16 07:56   3.5.1   server   0 1   Started throwing "Error Path:null Error:KeeperErrorCode = ReconfigInProgress" error when trying to change the cluster using reconfig and the IO hangged at one node.

Steps:-
1. Start Zookeeper in cluster mode
2. try to reconfig the cluster using "reconfig" command from one node's client (194) like
"reconfig -remove 3 -add 3=10.18.221.194:2888:3888;2181
3. make the IO busy for 5-10 secs at 194 node and then release
4. Again execute the above reconfig command
It is failing to execute even after 3-4 mins.
Server log is attached. (Complete server log is attached)

2016-09-02 18:12:05,845 [myid:2] - INFO [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):QuorumPeer@1074] - LEADING
2016-09-02 18:12:05,848 [myid:2] - INFO [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):Leader@63] - TCP NoDelay set to: true
2016-09-02 18:12:05,848 [myid:2] - INFO [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):Leader@83] - zookeeper.leader.maxConcurrentSnapshots = 10
2016-09-02 18:12:05,848 [myid:2] - INFO [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):Leader@85] - zookeeper.leader.maxConcurrentSnapshotTimeout = 5
2016-09-02 18:12:05,849 [myid:2] - INFO [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):ZooKeeperServer@858] - minSessionTimeout set to 4000
2016-09-02 18:12:05,849 [myid:2] - INFO [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):ZooKeeperServer@867] - maxSessionTimeout set to 40000
2016-09-02 18:12:05,849 [myid:2] - INFO [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):ZooKeeperServer@156] - Created server with tickTime 2000 minSessionTimeout 4000 maxSessionTimeout 40000 datadir /home/Rakesh/Zookeeper/18_Aug/cluster/zookeeper-3.5.1-alpha/data/version-2 snapdir /home/Rakesh/Zookeeper/18_Aug/cluster/zookeeper-3.5.1-alpha/data/version-2
2016-09-02 18:12:05,850 [myid:2] - INFO [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):Leader@412] - LEADING - LEADER ELECTION TOOK - 5
2016-09-02 18:12:05,852 [myid:2] - INFO [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):FileTxnSnapLog@298] - Snapshotting: 0x100000001 to /home/Rakesh/Zookeeper/18_Aug/cluster/zookeeper-3.5.1-alpha/data/version-2/snapshot.100000001
2016-09-02 18:12:06,854 [myid:2] - INFO [LearnerHandler-/10.18.101.80:55632:LearnerHandler@382] - Follower sid: 1 : info : 10.18.101.80:2888:3888:participant;0.0.0.0:2181
2016-09-02 18:12:06,869 [myid:2] - INFO [LearnerHandler-/10.18.101.80:55632:LearnerHandler@683] - Synchronizing with Follower sid: 1 maxCommittedLog=0x100000001 minCommittedLog=0x100000001 lastProcessedZxid=0x100000001 peerLastZxid=0x100000001
2016-09-02 18:12:06,869 [myid:2] - INFO [LearnerHandler-/10.18.101.80:55632:LearnerHandler@727] - Sending DIFF zxid=0x100000001 for peer sid: 1
2016-09-02 18:12:06,888 [myid:2] - INFO [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):Leader@1245] - Have quorum of supporters, sids: [ [1, 2] ]; starting up and setting last processed zxid: 0x200000000
2016-09-02 18:12:06,890 [myid:2] - INFO [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):CommitProcessor@254] - Configuring CommitProcessor with 8 worker threads.
2016-09-02 18:12:06,898 [myid:2] - INFO [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):ContainerManager@64] - Using checkIntervalMs=60000 maxPerMinute=10000
2016-09-02 18:12:18,886 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@688] - Notification: 2 (message format version), 3 (n.leader), 0x0 (n.zxid), 0xffffffffffffffff (n.round), LEADING (n.state), 3 (n.sid), 0x1 (n.peerEPoch), LEADING (my state)200000028 (n.config version)
2016-09-02 18:13:47,869 [myid:2] - INFO [ProcessThread(sid:2 cport:-1)::PrepRequestProcessor@512] - Incremental reconfig
2016-09-02 18:13:47,872 [myid:2] - ERROR [ProcessThread(sid:2 cport:-1)::QuorumPeer@1383] - setLastSeenQuorumVerifier called with stale config 8589934593. Current version: 8589934632
2016-09-02 18:14:15,545 [myid:2] - INFO [ProcessThread(sid:2 cport:-1)::PrepRequestProcessor@843] - Got user-level KeeperException when processing sessionid:0x1000aa5ce650000 type:reconfig cxid:0x3 zxid:0x200000002 txntype:-1 reqpath:n/a Error Path:null Error:KeeperErrorCode = ReconfigInProgress
2016-09-02 18:14:56,442 [myid:2] - INFO [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /10.18.219.50:48388
2016-09-02 18:14:56,454 [myid:2] - INFO [NIOWorkerThread-1:NIOServerCnxn@485] - Processing srvr command from /10.18.219.50:48388
2016-09-02 18:14:56,467 [myid:2] - INFO [NIOWorkerThread-1:NIOServerCnxn@606] - Closed socket connection for client /10.18.219.50:48388 (no session established for client)
2016-09-02 18:17:18,365 [myid:2] - INFO [ProcessThread(sid:2 cport:-1)::PrepRequestProcessor@843] - Got user-level KeeperException when processing sessionid:0x1000aa5ce650000 type:reconfig cxid:0x4 zxid:0x200000003 txntype:-1 reqpath:n/a Error Path:null Error:KeeperErrorCode = ReconfigInProgress
2016-09-02 18:19:23,699 [myid:2] - INFO [ProcessThread(sid:2 cport:-1)::PrepRequestProcessor@843] - Got user-level KeeperException when processing sessionid:0x1000aa5ce650000 type:reconfig cxid:0x5 zxid:0x200000004 txntype:-1 reqpath:n/a Error Path:null Error:KeeperErrorCode = ReconfigInProgress
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 28 weeks, 6 days ago 0|i336jz:
ZooKeeper ZOOKEEPER-2545

Keep maintaining the old zoo.cfg.dynamic* files which will keep eating system memory which is getting generated as part of reconfig execution

Bug Open Major Unresolved Rakesh Kumar Singh Rakesh Kumar Singh Rakesh Kumar Singh 02/Sep/16 02:53   23/Sep/16 01:08   3.5.1   server   0 1   Keep maintaining the old zoo.cfg.dynamic* files which will be getting created every time when "reconfig" is executed

Steps to reproduce:-1
1. Setup the zookeeper in cluster mode and start
2. trying running reconfig command like
>reconfig -remove 3 -add 1=10.18.101.80:2888:3888;2181
3. It will create new zoo.cfg.dynamic in conf folder
The problem is it is not deleting the old zoo.cfg.dynamic* files which will keep eating the memory
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 28 weeks, 6 days ago 0|i3362v:
ZooKeeper ZOOKEEPER-2544

" unary operator expected" message on the console while trying to get the zookeeper status

Bug Open Minor Unresolved Unassigned Rakesh Kumar Singh Rakesh Kumar Singh 02/Sep/16 01:45   08/Oct/17 12:41   3.5.1, 3.5.2   server   0 2   root@BLR1000010865:/home/Rakesh/Zookeeper/18_Aug/cluster/zookeeper-3.5.1-alpha/bin# ./zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /home/Rakesh/Zookeeper/18_Aug/cluster/zookeeper-3.5.1-alpha/bin/../conf/zoo.cfg
./zkServer.sh: line 223: [: =: unary operator expected
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
2 years, 23 weeks, 4 days ago 0|i3361b:
ZooKeeper ZOOKEEPER-2543

Add C API for creating nodes with TTL

New Feature Resolved Major Duplicate Unassigned Michael Han Michael Han 02/Sep/16 00:37   19/Dec/19 18:01 08/Oct/16 14:20 3.6.0   c client   0 1   ZOOKEEPER-2169 introduces a new feature for creation nodes with TTL, which is supported by Java client with new Java API. Similar API should be added to C client as well. ttl_nodes 9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 28 weeks, 6 days ago 0|i335z3:
ZooKeeper ZOOKEEPER-2542

Update NOTICE file with Netty notice in 3.4

Bug Closed Blocker Fixed Rakesh Radhakrishnan Flavio Paiva Junqueira Flavio Paiva Junqueira 31/Aug/16 13:40   31/Mar/17 05:01 12/Dec/16 18:04 3.4.9 3.4.10     0 4   We need to update the NOTICE file in the 3.4 branch as we did for 3.5 and trunk, see ZOOKEEPER-2459. newbie 9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 7 weeks, 2 days ago 0|i3331j:
ZooKeeper ZOOKEEPER-2541

We can't configure the "secureClientPort" in dynamicConfiguration and connect through client

Bug Open Major Unresolved Unassigned Rakesh Kumar Singh Rakesh Kumar Singh 30/Aug/16 01:58   30/Aug/16 01:58   3.5.1   server   1 3   We can't configure the "secureClientPort" in dynamicConfiguration and connect through client

Steps to reproduce:-
1. Configure the zookeeper in cluster mode with SSL mode
2. comment the clientport and secureClientport details from zoo.cfg file
3. Configure the secureClientport in dynamicConfiguration as below:-

server.1=10.18.101.80:2888:3888:participant;2181
server.2=10.18.219.50:2888:3888:participant;2181
server.3=10.18.221.194:2888:3888:participant;2181

4. Start the cluster
5. Start one client using zkCli.sh and try to connect to any one of the cluster, it fails

Client log:-
BLR1000007042:/home/Rakesh/Zookeeper/18_Aug/cluster/zookeeper-3.5.1-alpha/bin # ./zkCli.sh
/usr/bin/java
Connecting to localhost:2181
2016-08-30 13:42:33,574 [myid:] - INFO [main:Environment@109] - Client environment:zookeeper.version=3.5.1-alpha--1, built on 08/18/2016 08:20 GMT
2016-08-30 13:42:33,578 [myid:] - INFO [main:Environment@109] - Client environment:host.name=BLR1000007042
2016-08-30 13:42:33,578 [myid:] - INFO [main:Environment@109] - Client environment:java.version=1.7.0_79
2016-08-30 13:42:33,581 [myid:] - INFO [main:Environment@109] - Client environment:java.vendor=Oracle Corporation
2016-08-30 13:42:33,581 [myid:] - INFO [main:Environment@109] - Client environment:java.home=/usr/java/jdk1.7.0_79/jre
2016-08-30 13:42:33,581 [myid:] - INFO [main:Environment@109] - Client environment:java.class.path=/home/Rakesh/Zookeeper/18_Aug/cluster/zookeeper-3.5.1-alpha/bin/../build/classes:/home/Rakesh/Zookeeper/18_Aug/cluster/zookeeper-3.5.1-alpha/bin/../build/lib/*.jar:/home/Rakesh/Zookeeper/18_Aug/cluster/zookeeper-3.5.1-alpha/bin/../lib/slf4j-log4j12-1.7.5.jar:/home/Rakesh/Zookeeper/18_Aug/cluster/zookeeper-3.5.1-alpha/bin/../lib/slf4j-api-1.7.5.jar:/home/Rakesh/Zookeeper/18_Aug/cluster/zookeeper-3.5.1-alpha/bin/../lib/servlet-api-2.5-20081211.jar:/home/Rakesh/Zookeeper/18_Aug/cluster/zookeeper-3.5.1-alpha/bin/../lib/netty-3.7.0.Final.jar:/home/Rakesh/Zookeeper/18_Aug/cluster/zookeeper-3.5.1-alpha/bin/../lib/log4j-1.2.16.jar:/home/Rakesh/Zookeeper/18_Aug/cluster/zookeeper-3.5.1-alpha/bin/../lib/jline-2.11.jar:/home/Rakesh/Zookeeper/18_Aug/cluster/zookeeper-3.5.1-alpha/bin/../lib/jetty-util-6.1.26.jar:/home/Rakesh/Zookeeper/18_Aug/cluster/zookeeper-3.5.1-alpha/bin/../lib/jetty-6.1.26.jar:/home/Rakesh/Zookeeper/18_Aug/cluster/zookeeper-3.5.1-alpha/bin/../lib/javacc.jar:/home/Rakesh/Zookeeper/18_Aug/cluster/zookeeper-3.5.1-alpha/bin/../lib/jackson-mapper-asl-1.9.11.jar:/home/Rakesh/Zookeeper/18_Aug/cluster/zookeeper-3.5.1-alpha/bin/../lib/jackson-core-asl-1.9.11.jar:/home/Rakesh/Zookeeper/18_Aug/cluster/zookeeper-3.5.1-alpha/bin/../lib/commons-cli-1.2.jar:/home/Rakesh/Zookeeper/18_Aug/cluster/zookeeper-3.5.1-alpha/bin/../lib/ant-eclipse-1.0-jvm1.2.jar:/home/Rakesh/Zookeeper/18_Aug/cluster/zookeeper-3.5.1-alpha/bin/../zookeeper-3.5.1-alpha.jar:/home/Rakesh/Zookeeper/18_Aug/cluster/zookeeper-3.5.1-alpha/bin/../src/java/lib/ant-eclipse-1.0-jvm1.2.jar:/home/Rakesh/Zookeeper/18_Aug/cluster/zookeeper-3.5.1-alpha/bin/../conf:
2016-08-30 13:42:33,582 [myid:] - INFO [main:Environment@109] - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2016-08-30 13:42:33,582 [myid:] - INFO [main:Environment@109] - Client environment:java.io.tmpdir=/tmp
2016-08-30 13:42:33,582 [myid:] - INFO [main:Environment@109] - Client environment:java.compiler=<NA>
2016-08-30 13:42:33,582 [myid:] - INFO [main:Environment@109] - Client environment:os.name=Linux
2016-08-30 13:42:33,582 [myid:] - INFO [main:Environment@109] - Client environment:os.arch=amd64
2016-08-30 13:42:33,583 [myid:] - INFO [main:Environment@109] - Client environment:os.version=3.0.76-0.11-default
2016-08-30 13:42:33,583 [myid:] - INFO [main:Environment@109] - Client environment:user.name=root
2016-08-30 13:42:33,583 [myid:] - INFO [main:Environment@109] - Client environment:user.home=/root
2016-08-30 13:42:33,583 [myid:] - INFO [main:Environment@109] - Client environment:user.dir=/home/Rakesh/Zookeeper/18_Aug/cluster/zookeeper-3.5.1-alpha/bin
2016-08-30 13:42:33,583 [myid:] - INFO [main:Environment@109] - Client environment:os.memory.free=52MB
2016-08-30 13:42:33,586 [myid:] - INFO [main:Environment@109] - Client environment:os.memory.max=227MB
2016-08-30 13:42:33,587 [myid:] - INFO [main:Environment@109] - Client environment:os.memory.total=57MB
2016-08-30 13:42:33,591 [myid:] - INFO [main:ZooKeeper@716] - Initiating client connection, connectString=localhost:2181 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@272f15b0
Welcome to ZooKeeper!
2016-08-30 13:42:33,681 [myid:] - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread@1138] - Opening socket connection to server localhost/0:0:0:0:0:0:0:1:2181. Will not attempt to authenticate using SASL (unknown error)
JLine support is enabled
[INFO] Unable to bind key for unsupported operation: backward-delete-word
[INFO] Unable to bind key for unsupported operation: backward-delete-word
[INFO] Unable to bind key for unsupported operation: down-history
[INFO] Unable to bind key for unsupported operation: up-history
[INFO] Unable to bind key for unsupported operation: up-history
[INFO] Unable to bind key for unsupported operation: down-history
[INFO] Unable to bind key for unsupported operation: up-history
[INFO] Unable to bind key for unsupported operation: down-history
[INFO] Unable to bind key for unsupported operation: up-history
[INFO] Unable to bind key for unsupported operation: down-history
[INFO] Unable to bind key for unsupported operation: up-history
[INFO] Unable to bind key for unsupported operation: down-history
[zk: localhost:2181(CONNECTING) 0] 2016-08-30 13:42:33,975 [myid:] - INFO [main-SendThread(localhost:2181):ClientCnxnSocketNetty$ZKClientPipelineFactory@363] - SSL handler added for channel: null
2016-08-30 13:42:34,004 [myid:] - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread@980] - Socket connection established, initiating session, client: /0:0:0:0:0:0:0:1:47374, server: localhost/0:0:0:0:0:0:0:1:2181
2016-08-30 13:42:34,006 [myid:] - INFO [main-SendThread(localhost:2181):ClientCnxnSocketNetty$1@146] - channel is connected: [id: 0xd4aaee7b, /0:0:0:0:0:0:0:1:47374 => localhost/0:0:0:0:0:0:0:1:2181]
2016-08-30 13:42:34,030 [myid:] - INFO [New I/O worker #1:ClientCnxnSocketNetty$ZKClientHandler@377] - channel is disconnected: [id: 0xd4aaee7b, /0:0:0:0:0:0:0:1:47374 :> localhost/0:0:0:0:0:0:0:1:2181]
2016-08-30 13:42:34,030 [myid:] - INFO [New I/O worker #1:ClientCnxnSocketNetty@201] - channel is told closing
2016-08-30 13:42:34,030 [myid:] - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread@1253] - channel for sessionid 0x0 is lost, closing socket connection and attempting reconnect
2016-08-30 13:42:34,033 [myid:] - WARN [New I/O worker #1:ClientCnxnSocketNetty$ZKClientHandler@432] - Exception caught: [id: 0xd4aaee7b, /0:0:0:0:0:0:0:1:47374 :> localhost/0:0:0:0:0:0:0:1:2181] EXCEPTION: java.nio.channels.ClosedChannelException
java.nio.channels.ClosedChannelException
at org.jboss.netty.handler.ssl.SslHandler$6.run(SslHandler.java:1580)
at org.jboss.netty.channel.socket.ChannelRunnableWrapper.run(ChannelRunnableWrapper.java:40)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.executeInIoThread(AbstractNioWorker.java:71)
at org.jboss.netty.channel.socket.nio.NioWorker.executeInIoThread(NioWorker.java:36)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.executeInIoThread(AbstractNioWorker.java:57)
at org.jboss.netty.channel.socket.nio.NioWorker.executeInIoThread(NioWorker.java:36)
at org.jboss.netty.channel.socket.nio.AbstractNioChannelSink.execute(AbstractNioChannelSink.java:34)
at org.jboss.netty.handler.ssl.SslHandler.channelClosed(SslHandler.java:1566)
at org.jboss.netty.channel.Channels.fireChannelClosed(Channels.java:468)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:376)
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:93)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:109)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:90)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2016-08-30 13:42:34,230 [myid:] - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread@1138] - Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
2016-08-30 13:42:34,240 [myid:] - INFO [main-SendThread(localhost:2181):ClientCnxnSocketNetty$ZKClientPipelineFactory@363] - SSL handler added for channel: null
2016-08-30 13:42:34,241 [myid:] - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread@980] - Socket connection established, initiating session, client: /127.0.0.1:60295, server: localhost/127.0.0.1:2181


Server log (It is not starting in secureMode though all the required configuration is done for ssl except secureClientport which is configured in dynamicConfiguration):-
2016-08-30 13:40:13,436 [myid:1] - INFO [QuorumPeer[myid=1](plain=/0.0.0.0:2181)(secure=disabled):FastLeaderElection@928] - Notification time out: 800
2016-08-30 13:40:14,239 [myid:1] - WARN [QuorumPeer[myid=1](plain=/0.0.0.0:2181)(secure=disabled):QuorumCnxManager@459] - Cannot open channel to 2 at election address /10.18.219.50:3888
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:444)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:485)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:513)
at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:919)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1040)
201
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 29 weeks, 2 days ago 0|i32zlb:
ZooKeeper ZOOKEEPER-2540

When start zookeeper server by configuring the server details in dynamic configuration with passing the client port, wrong log info is logged

Bug Open Minor Unresolved Rakesh Kumar Singh Rakesh Kumar Singh Rakesh Kumar Singh 30/Aug/16 01:16   23/Sep/16 01:09   3.5.1   server   0 1   When start zookeeper server by configuring the server details in dynamic configuration with passing the client port, wrong log info is logged:-

Configure the server details as below which contains client port as well and remove the client port from zoo.cfg (as it is duplicate) :-
server.1=10.18.101.80:2888:3888:participant;2181
server.2=10.18.219.50:2888:3888:participant;2181
server.3=10.18.221.194:2888:3888:participant;2181

Start the cluster, we can see message as

2016-08-30 17:00:33,984 [myid:] - INFO [main:QuorumPeerConfig@306] - clientPort is not set

which is not correct
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 29 weeks, 2 days ago 0|i32zjz:
ZooKeeper ZOOKEEPER-2539

Throwing nullpointerException when run the command "config -c" when client port is mentioned as separate and not like new style

Bug Closed Minor Fixed Rakesh Kumar Singh Rakesh Kumar Singh Rakesh Kumar Singh 29/Aug/16 10:09   17/May/17 23:43 08/Sep/16 10:22 3.5.1, 3.5.2 3.5.3, 3.6.0 java client   0 4   Throwing nullpointerException when run the command "config -c" when client port is mentioned as separate and not like new style
1. Configure the zookeeper to start in cluster mode like below-

clientPort=2181

server.1=10.18.101.80:2888:3888
server.2=10.18.219.50:2888:3888
server.3=10.18.221.194:2888:3888

and not like below:-
server.1=10.18.101.80:2888:3888:participant;2181
server.2=10.18.219.50:2888:3888:participant;2181
server.3=10.18.221.194:2888:3888:participant;2181

2. Start the cluster and one client using >zkCli.sh
3. execute command "config -c"
It is throwing nullpointerException:-


root@BLR1000010865:/home/Rakesh/Zookeeper/18_Aug/cluster/zookeeper-3.5.1-alpha/bin# ./zkCli.sh
Connecting to localhost:2181
2016-08-29 21:45:19,558 [myid:] - INFO [main:Environment@109] - Client environment:zookeeper.version=3.5.1-alpha--1, built on 08/18/2016 08:20 GMT
2016-08-29 21:45:19,561 [myid:] - INFO [main:Environment@109] - Client environment:host.name=BLR1000010865
2016-08-29 21:45:19,562 [myid:] - INFO [main:Environment@109] - Client environment:java.version=1.7.0_17
2016-08-29 21:45:19,564 [myid:] - INFO [main:Environment@109] - Client environment:java.vendor=Oracle Corporation
2016-08-29 21:45:19,564 [myid:] - INFO [main:Environment@109] - Client environment:java.home=/usr/lib/jvm/oracle_jdk7/jre
2016-08-29 21:45:19,564 [myid:] - INFO [main:Environment@109] - Client environment:java.class.path=/home/Rakesh/Zookeeper/18_Aug/cluster/zookeeper-3.5.1-alpha/bin/../build/classes:/home/Rakesh/Zookeeper/18_Aug/cluster/zookeeper-3.5.1-alpha/bin/../build/lib/*.jar:/home/Rakesh/Zookeeper/18_Aug/cluster/zookeeper-3.5.1-alpha/bin/../lib/slf4j-log4j12-1.7.5.jar:/home/Rakesh/Zookeeper/18_Aug/cluster/zookeeper-3.5.1-alpha/bin/../lib/slf4j-api-1.7.5.jar:/home/Rakesh/Zookeeper/18_Aug/cluster/zookeeper-3.5.1-alpha/bin/../lib/servlet-api-2.5-20081211.jar:/home/Rakesh/Zookeeper/18_Aug/cluster/zookeeper-3.5.1-alpha/bin/../lib/netty-3.7.0.Final.jar:/home/Rakesh/Zookeeper/18_Aug/cluster/zookeeper-3.5.1-alpha/bin/../lib/log4j-1.2.16.jar:/home/Rakesh/Zookeeper/18_Aug/cluster/zookeeper-3.5.1-alpha/bin/../lib/jline-2.11.jar:/home/Rakesh/Zookeeper/18_Aug/cluster/zookeeper-3.5.1-alpha/bin/../lib/jetty-util-6.1.26.jar:/home/Rakesh/Zookeeper/18_Aug/cluster/zookeeper-3.5.1-alpha/bin/../lib/jetty-6.1.26.jar:/home/Rakesh/Zookeeper/18_Aug/cluster/zookeeper-3.5.1-alpha/bin/../lib/javacc.jar:/home/Rakesh/Zookeeper/18_Aug/cluster/zookeeper-3.5.1-alpha/bin/../lib/jackson-mapper-asl-1.9.11.jar:/home/Rakesh/Zookeeper/18_Aug/cluster/zookeeper-3.5.1-alpha/bin/../lib/jackson-core-asl-1.9.11.jar:/home/Rakesh/Zookeeper/18_Aug/cluster/zookeeper-3.5.1-alpha/bin/../lib/commons-cli-1.2.jar:/home/Rakesh/Zookeeper/18_Aug/cluster/zookeeper-3.5.1-alpha/bin/../lib/ant-eclipse-1.0-jvm1.2.jar:/home/Rakesh/Zookeeper/18_Aug/cluster/zookeeper-3.5.1-alpha/bin/../zookeeper-3.5.1-alpha.jar:/home/Rakesh/Zookeeper/18_Aug/cluster/zookeeper-3.5.1-alpha/bin/../src/java/lib/ant-eclipse-1.0-jvm1.2.jar:/home/Rakesh/Zookeeper/18_Aug/cluster/zookeeper-3.5.1-alpha/bin/../conf:
2016-08-29 21:45:19,564 [myid:] - INFO [main:Environment@109] - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2016-08-29 21:45:19,564 [myid:] - INFO [main:Environment@109] - Client environment:java.io.tmpdir=/tmp
2016-08-29 21:45:19,564 [myid:] - INFO [main:Environment@109] - Client environment:java.compiler=<NA>
2016-08-29 21:45:19,565 [myid:] - INFO [main:Environment@109] - Client environment:os.name=Linux
2016-08-29 21:45:19,565 [myid:] - INFO [main:Environment@109] - Client environment:os.arch=amd64
2016-08-29 21:45:19,565 [myid:] - INFO [main:Environment@109] - Client environment:os.version=4.4.0-31-generic
2016-08-29 21:45:19,565 [myid:] - INFO [main:Environment@109] - Client environment:user.name=root
2016-08-29 21:45:19,565 [myid:] - INFO [main:Environment@109] - Client environment:user.home=/root
2016-08-29 21:45:19,565 [myid:] - INFO [main:Environment@109] - Client environment:user.dir=/home/Rakesh/Zookeeper/18_Aug/cluster/zookeeper-3.5.1-alpha/bin
2016-08-29 21:45:19,565 [myid:] - INFO [main:Environment@109] - Client environment:os.memory.free=114MB
2016-08-29 21:45:19,567 [myid:] - INFO [main:Environment@109] - Client environment:os.memory.max=227MB
2016-08-29 21:45:19,568 [myid:] - INFO [main:Environment@109] - Client environment:os.memory.total=119MB
2016-08-29 21:45:19,570 [myid:] - INFO [main:ZooKeeper@716] - Initiating client connection, connectString=localhost:2181 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@149ee0f1
Welcome to ZooKeeper!
2016-08-29 21:45:19,596 [myid:] - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread@1138] - Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
2016-08-29 21:45:19,603 [myid:] - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread@980] - Socket connection established, initiating session, client: /127.0.0.1:43574, server: localhost/127.0.0.1:2181
JLine support is enabled
2016-08-29 21:45:19,630 [myid:] - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread@1400] - Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x20044a0c51d0000, negotiated timeout = 30000

WATCHER::

WatchedEvent state:SyncConnected type:None path:null
[zk: localhost:2181(CONNECTED) 0]
[zk: localhost:2181(CONNECTED) 0] config -c
Exception in thread "main" java.lang.NullPointerException
at org.apache.zookeeper.server.util.ConfigUtils.getClientConfigStr(ConfigUtils.java:56)
at org.apache.zookeeper.cli.GetConfigCommand.exec(GetConfigCommand.java:64)
at org.apache.zookeeper.ZooKeeperMain.processZKCmd(ZooKeeperMain.java:674)
at org.apache.zookeeper.ZooKeeperMain.processCmd(ZooKeeperMain.java:577)
at org.apache.zookeeper.ZooKeeperMain.executeLine(ZooKeeperMain.java:360)
at org.apache.zookeeper.ZooKeeperMain.run(ZooKeeperMain.java:320)
at org.apache.zookeeper.ZooKeeperMain.main(ZooKeeperMain.java:280)
root@BLR1000010865:/home/Rakesh/Zookeeper/18_Aug/cluster/zookeeper-3.5.1-alpha/bin#
9223372036854775807 No Perforce job exists for this issue. 3 9223372036854775807
3 years, 28 weeks ago
Reviewed
0|i32y8v:
ZooKeeper ZOOKEEPER-2538

ZOOKEEPER-3170 Flaky Test: org.apache.zookeeper.server.quorum.Zab1_0Test.testNormalObserverRun

Sub-task Closed Major Cannot Reproduce Andor Molnar Michael Han Michael Han 26/Aug/16 17:15   19/Dec/19 18:01 25/Oct/18 10:46 3.5.2 3.5.5 quorum, server, tests   0 3   The failure of this test is fairly often. Might relate to ZOOKEEPER-1798.

{noformat}
org.apache.zookeeper.server.quorum.Zab1_0Test.testNormalObserverRun

Failing for the past 1 build (Since Failed#1143 )
Took 0 ms.
Error Message

Timeout occurred. Please note the time in the report does not reflect the time until the timeout.
Stacktrace

junit.framework.AssertionFailedError: Timeout occurred. Please note the time in the report does not reflect the time until the timeout.
at java.lang.Thread.run(Thread.java:745)
{noformat}
flaky, flaky-test 9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
1 year, 21 weeks ago 0|i32vd3:
ZooKeeper ZOOKEEPER-2537

When provide path for "dataDir" with heading space, it is taking correct path (by trucating space) for snapshot but zookeeper_server.pid is getting created in root (/) folder

Bug Closed Major Duplicate Rakesh Kumar Singh Rakesh Kumar Singh Rakesh Kumar Singh 26/Aug/16 05:08   19/Dec/19 18:02 12/Sep/16 18:13 3.5.1, 3.5.2 3.5.3 server   0 2   Scenario 1 :-
When provide path for "dataDir" with heading space, it is taking correct path (by trucating space) for snapshot but zookeeper_server.pid is getting created in root (/) folder
Steps to reproduce:-
1. Configure the dataDir
dataDir= /home/Rakesh/Zookeeper/18_Aug/zookeeper-3.5.1-alpha/data
Here there is a space after dataDir=
2. Start Zookeeper Server
3. The snapshot is getting created at location mentioned above by truncating the heading space but
zookeeper_server.pid is getting created at root (/) folder
9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
3 years, 27 weeks, 3 days ago 0|i32u4n:
ZooKeeper ZOOKEEPER-2536

When provide path for "dataDir" with trailing space, it is taking correct path (by trucating space) for snapshot but creating temporary file with some junk folder name for zookeeper_server.pid

Bug Closed Major Fixed Rakesh Kumar Singh Rakesh Kumar Singh Rakesh Kumar Singh 26/Aug/16 04:57   17/May/17 23:43 08/Sep/16 00:07 3.5.1, 3.5.2 3.5.3, 3.6.0 server   0 4   Scenario 1:-
When provide path for "dataDir" with trailing space, it is taking correct path (by trucating space) for snapshot but creating temporary file with some junk folder name for zookeeper_server.pid

Steps to reproduce:-
1. Configure the dataDir
dataDir=/home/Rakesh/Zookeeper/18_Aug/zookeeper-3.5.1-alpha/data
Here there is a space after /data
2. Start Zookeeper Server
3. The snapshot is getting created at location mentioned above by truncating the trailing space but
one temp folder with junk name (like -> D29D4X~J) is getting created for zookeeper_server.pid

Scenario 2:-
When configure the heading and trailing space in above mentioned scenario. the temp folder is getting created in zookeeper/bin folder
9223372036854775807 No Perforce job exists for this issue. 2 9223372036854775807
3 years, 28 weeks ago
Reviewed
0|i32u3r:
ZooKeeper ZOOKEEPER-2535

zookeeper_server's pid should be created once server is started completely and not before, problem with current approach

Bug Open Major Unresolved Unassigned Rakesh Kumar Singh Rakesh Kumar Singh 26/Aug/16 02:58   26/Aug/16 03:00   3.5.1   server   0 1   zookeeper_server's pid should be created once server is started completely and not before, problem with current approach :-
Scenario:-
1. Configure below in zoo.cfg
dataDir=/home/Rakesh/Zookeeper/18_Aug/zookeeper-3.5.1-alpha/data
2. Start zookeeper server
3. Change the dataDir to suppose
dataDir=/home/Rakesh/Zookeeper/18_Aug/zookeeper-3.5.1-alpha/data1
4. Again start zookeeper without stopping zookeeper.

Though it will fail to start the server as port is already bind, it will create "zookeeper_server.pid" file with different PID inside and "version-2" folder

Now again revertback the dataDir path and stop the server, the new created folder and file at step 4 remained
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 29 weeks, 6 days ago 0|i32u0f:
ZooKeeper ZOOKEEPER-2534

Configuration for IP address (zookeeper.server.ip=) should be introduced and zookeeper server should bind to that particular IP only not to all IPs of the system

Bug Open Major Unresolved Unassigned Rakesh Kumar Singh Rakesh Kumar Singh 26/Aug/16 01:46   26/Aug/16 01:47   3.5.1, 3.5.2   security   0 1   Configuration for IP address (zookeeper.server.ip=) should be introduced and zookeeper server should bind to that particular IP only not to all IPs of the system. As of now zookeeper is binding to 0.0.0.0 (all IPs - 127.0.0.1, IPv4, IPv6) of the system. 9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 29 weeks, 6 days ago 0|i32tx3:
ZooKeeper ZOOKEEPER-2533

Close the zkCli using "close" command and then connect using "connect" then provide some invalid input, it closing the channel and connecting again

Bug Open Minor Unresolved Rakesh Kumar Singh Rakesh Kumar Singh Rakesh Kumar Singh 25/Aug/16 05:16   23/Sep/16 01:10   3.5.1   java client   0 1   Close the zkCli using "close" command and then connect using "connect" then provide some invalid input, it closing the channel and connecting again

Steps to reproduce:-
1. Connect the Zookeeper server using zkCli
2. close the connection using "close"
3. Connect again using "connect host"
4. Once connected, input space " " and hit enter
It is closing the channel and establishing again.
Console log is as below:-

[zk: localhost:2181(CONNECTED) 5] close
2016-08-25 16:59:04,854 [myid:] - INFO [main:ClientCnxnSocketNetty@201] - channel is told closing
2016-08-25 16:59:04,855 [myid:] - INFO [main:ZooKeeper@1110] - Session: 0x101a00305cc0008 closed
[zk: localhost:2181(CLOSED) 6] 2016-08-25 16:59:04,855 [myid:] - INFO [main-EventThread:ClientCnxn$EventThread@542] - EventThread shut down for session: 0x101a00305cc0008
2016-08-25 16:59:04,856 [myid:] - INFO [New I/O worker #1:ClientCnxnSocketNetty$ZKClientHandler@377] - channel is disconnected: [id: 0xd9735868, /0:0:0:0:0:0:0:1:44595 :> localhost/0:0:0:0:0:0:0:1:2181]
2016-08-25 16:59:04,856 [myid:] - INFO [New I/O worker #1:ClientCnxnSocketNetty@201] - channel is told closing
connect 10.18.101.80
2016-08-25 16:59:14,410 [myid:] - INFO [main:ZooKeeper@716] - Initiating client connection, connectString=10.18.101.80 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@19c50523
[zk: 10.18.101.80(CONNECTING) 7] 2016-08-25 16:59:14,417 [myid:] - INFO [main-SendThread(10.18.101.80:2181):ClientCnxn$SendThread@1138] - Opening socket connection to server 10.18.101.80/10.18.101.80:2181. Will not attempt to authenticate using SASL (unknown error)
2016-08-25 16:59:14,426 [myid:] - INFO [main-SendThread(10.18.101.80:2181):ClientCnxnSocketNetty$ZKClientPipelineFactory@363] - SSL handler added for channel: null
2016-08-25 16:59:14,428 [myid:] - INFO [New I/O worker #10:ClientCnxn$SendThread@980] - Socket connection established, initiating session, client: /10.18.101.80:58871, server: 10.18.101.80/10.18.101.80:2181
2016-08-25 16:59:14,428 [myid:] - INFO [New I/O worker #10:ClientCnxnSocketNetty$1@146] - channel is connected: [id: 0xa8f6b724, /10.18.101.80:58871 => 10.18.101.80/10.18.101.80:2181]
2016-08-25 16:59:14,473 [myid:] - INFO [New I/O worker #10:ClientCnxn$SendThread@1400] - Session establishment complete on server 10.18.101.80/10.18.101.80:2181, sessionid = 0x101a00305cc0009, negotiated timeout = 30000

WATCHER::

WatchedEvent state:SyncConnected type:None path:null
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 30 weeks ago 0|i32rxr:
ZooKeeper ZOOKEEPER-2532

zkCli throwing nullpointerException instead it should provide help when invalid input is entered

Bug Patch Available Minor Unresolved Rakesh Kumar Singh Rakesh Kumar Singh Rakesh Kumar Singh 25/Aug/16 05:11   05/Feb/20 07:11   3.5.1, 3.5.2 3.7.0, 3.5.8 java client   1 4   1. a) Connect to zookeeper using zkCli
b) just input space and then hit enter

2. a) Connect to zookeeper using zkCli and hit enter it will come as connected
b) just input space and then hit enter

Console log is as below:-
[zk: localhost:2181(CONNECTING) 0] 2016-08-25 16:54:48,143 [myid:] - INFO [main-SendThread(localhost:2181):ClientCnxnSocketNetty$ZKClientPipelineFactory@363] - SSL handler added for channel: null
2016-08-25 16:54:48,175 [myid:] - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread@980] - Socket connection established, initiating session, client: /0:0:0:0:0:0:0:1:44592, server: localhost/0:0:0:0:0:0:0:1:2181
2016-08-25 16:54:48,178 [myid:] - INFO [main-SendThread(localhost:2181):ClientCnxnSocketNetty$1@146] - channel is connected: [id: 0xd03f4226, /0:0:0:0:0:0:0:1:44592 => localhost/0:0:0:0:0:0:0:1:2181]
2016-08-25 16:54:48,288 [myid:] - INFO [New I/O worker #1:ClientCnxn$SendThread@1400] - Session establishment complete on server localhost/0:0:0:0:0:0:0:1:2181, sessionid = 0x101a00305cc0005, negotiated timeout = 30000

WATCHER::

WatchedEvent state:SyncConnected type:None path:null

Exception in thread "main" java.lang.NullPointerException
at org.apache.zookeeper.ZooKeeperMain$MyCommandOptions.getArgArray(ZooKeeperMain.java:171)
at org.apache.zookeeper.ZooKeeperMain.processZKCmd(ZooKeeperMain.java:613)
at org.apache.zookeeper.ZooKeeperMain.processCmd(ZooKeeperMain.java:577)
at org.apache.zookeeper.ZooKeeperMain.executeLine(ZooKeeperMain.java:360)
at org.apache.zookeeper.ZooKeeperMain.run(ZooKeeperMain.java:320)
at org.apache.zookeeper.ZooKeeperMain.main(ZooKeeperMain.java:280)



----------------------------------------
After connection is established, input space and hit enter


[zk: localhost:2181(CONNECTING) 0] 2016-08-25 16:56:22,445 [myid:] - INFO [main-SendThread(localhost:2181):ClientCnxnSocketNetty$ZKClientPipelineFactory@363] - SSL handler added for channel: null
2016-08-25 16:56:22,481 [myid:] - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread@980] - Socket connection established, initiating session, client: /0:0:0:0:0:0:0:1:44594, server: localhost/0:0:0:0:0:0:0:1:2181
2016-08-25 16:56:22,484 [myid:] - INFO [main-SendThread(localhost:2181):ClientCnxnSocketNetty$1@146] - channel is connected: [id: 0xe6d3a461, /0:0:0:0:0:0:0:1:44594 => localhost/0:0:0:0:0:0:0:1:2181]
2016-08-25 16:56:22,597 [myid:] - INFO [New I/O worker #1:ClientCnxn$SendThread@1400] - Session establishment complete on server localhost/0:0:0:0:0:0:0:1:2181, sessionid = 0x101a00305cc0007, negotiated timeout = 30000

WATCHER::

WatchedEvent state:SyncConnected type:None path:null

[zk: localhost:2181(CONNECTED) 0]
Exception in thread "main" java.lang.NullPointerException
at org.apache.zookeeper.ZooKeeperMain$MyCommandOptions.getArgArray(ZooKeeperMain.java:171)
at org.apache.zookeeper.ZooKeeperMain.processZKCmd(ZooKeeperMain.java:613)
at org.apache.zookeeper.ZooKeeperMain.processCmd(ZooKeeperMain.java:577)
at org.apache.zookeeper.ZooKeeperMain.executeLine(ZooKeeperMain.java:360)
at org.apache.zookeeper.ZooKeeperMain.run(ZooKeeperMain.java:320)
at org.apache.zookeeper.ZooKeeperMain.main(ZooKeeperMain.java:280)
patch 9223372036854775807 No Perforce job exists for this issue. 2 9223372036854775807
3 years, 25 weeks, 3 days ago submitting the patch again 0|i32rx3:
ZooKeeper ZOOKEEPER-2531

Configuration as "zookeeper.secure=true/false" can be introduced and reading and verifying all ssl related configuration (like secureport, keystore, truststore, corresponding password) should be done only when "zookeeper.secure" flag is true

Bug Open Major Unresolved Unassigned Rakesh Kumar Singh Rakesh Kumar Singh 25/Aug/16 02:58   16/Sep/16 07:36   3.5.1   server   0 1   Configuration as "zookeeper.secure=true/false" can be introduced and reading and verifying all ssl related configuration (like secureport, keystore, truststore, corresponding password) should be done only when "zookeeper.secure" flag is true 9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 30 weeks ago 0|i32rpz:
ZooKeeper ZOOKEEPER-2530

When zookeeper started in SSL mode, set a "get" watcher on a znode from zkCli client, restart the zkCli, the "Data" watcher still present. Trying removing that watcher fails saying no watcher available

Bug Open Major Unresolved Unassigned Rakesh Kumar Singh Rakesh Kumar Singh 25/Aug/16 02:40   28/Jun/17 03:08   3.5.1   server   0 2   When zookeeper started in SSL mode, set a "get" watcher on a znode from zkCli client, restart the zkCli, the "Data" watcher still present. Trying removing that watcher fails saying no watcher available

Steps to reproduce:-

Start Zookeeper server in ssl mode by configuring all required ssl configuration
Start zkCli and set a "Data" watcher "get -w "
Restart the zkCli client
Check the watcher. Still the Data watcher is available
Try to remove the watcher using removewachers, it fails saying no watcher available.
BLR1000007042:~ # echo wchs | netcat localhost 3181
1 connections watching 1 paths
Total watches:1
BLR1000007042:~ # echo wchs | netcat localhost 3181
1 connections watching 1 paths
Total watches:1

Client log as below:-

[zk: localhost:2181(CONNECTED) 0] get -w /test
hello1
[zk: localhost:2181(CONNECTED) 1] quit
2016-08-25 14:22:00,706 [myid:] - INFO [main:ClientCnxnSocketNetty@201] - channel is told closing
2016-08-25 14:22:00,706 [myid:] - INFO [main:ZooKeeper@1110] - Session: 0x1019f8940e20000 closed
2016-08-25 14:22:00,706 [myid:] - INFO [main-EventThread:ClientCnxn$EventThread@542] - EventThread shut down for session: 0x1019f8940e20000
2016-08-25 14:22:00,707 [myid:] - INFO [New I/O worker #1:ClientCnxnSocketNetty$ZKClientHandler@377] - channel is disconnected: [id: 0x9dab735e, /127.0.0.1:57415 :> localhost/127.0.0.1:2181]
2016-08-25 14:22:00,707 [myid:] - INFO [New I/O worker #1:ClientCnxnSocketNetty@201] - channel is told closing
BLR1000007042:/home/Rakesh/Zookeeper/18_Aug/zookeeper-3.5.1-alpha/bin # ./zkCli.sh
/usr/bin/java
Connecting to localhost:2181
2016-08-25 14:22:15,079 [myid:] - INFO [main:Environment@109] - Client environment:zookeeper.version=3.5.1-alpha--1, built on 08/18/2016 08:20 GMT
2016-08-25 14:22:15,083 [myid:] - INFO [main:Environment@109] - Client environment:host.name=BLR1000007042
2016-08-25 14:22:15,084 [myid:] - INFO [main:Environment@109] - Client environment:java.version=1.7.0_79
2016-08-25 14:22:15,086 [myid:] - INFO [main:Environment@109] - Client environment:java.vendor=Oracle Corporation
2016-08-25 14:22:15,086 [myid:] - INFO [main:Environment@109] - Client environment:java.home=/usr/java/jdk1.7.0_79/jre
2016-08-25 14:22:15,086 [myid:] - INFO [main:Environment@109] - Client environment:java.class.path=/home/Rakesh/Zookeeper/18_Aug/zookeeper-3.5.1-alpha/bin/../build/classes:/home/Rakesh/Zookeeper/18_Aug/zookeeper-3.5.1-alpha/bin/../build/lib/*.jar:/home/Rakesh/Zookeeper/18_Aug/zookeeper-3.5.1-alpha/bin/../lib/slf4j-log4j12-1.7.5.jar:/home/Rakesh/Zookeeper/18_Aug/zookeeper-3.5.1-alpha/bin/../lib/slf4j-api-1.7.5.jar:/home/Rakesh/Zookeeper/18_Aug/zookeeper-3.5.1-alpha/bin/../lib/servlet-api-2.5-20081211.jar:/home/Rakesh/Zookeeper/18_Aug/zookeeper-3.5.1-alpha/bin/../lib/netty-3.7.0.Final.jar:/home/Rakesh/Zookeeper/18_Aug/zookeeper-3.5.1-alpha/bin/../lib/log4j-1.2.16.jar:/home/Rakesh/Zookeeper/18_Aug/zookeeper-3.5.1-alpha/bin/../lib/jline-2.11.jar:/home/Rakesh/Zookeeper/18_Aug/zookeeper-3.5.1-alpha/bin/../lib/jetty-util-6.1.26.jar:/home/Rakesh/Zookeeper/18_Aug/zookeeper-3.5.1-alpha/bin/../lib/jetty-6.1.26.jar:/home/Rakesh/Zookeeper/18_Aug/zookeeper-3.5.1-alpha/bin/../lib/javacc.jar:/home/Rakesh/Zookeeper/18_Aug/zookeeper-3.5.1-alpha/bin/../lib/jackson-mapper-asl-1.9.11.jar:/home/Rakesh/Zookeeper/18_Aug/zookeeper-3.5.1-alpha/bin/../lib/jackson-core-asl-1.9.11.jar:/home/Rakesh/Zookeeper/18_Aug/zookeeper-3.5.1-alpha/bin/../lib/commons-cli-1.2.jar:/home/Rakesh/Zookeeper/18_Aug/zookeeper-3.5.1-alpha/bin/../lib/ant-eclipse-1.0-jvm1.2.jar:/home/Rakesh/Zookeeper/18_Aug/zookeeper-3.5.1-alpha/bin/../zookeeper-3.5.1-alpha.jar:/home/Rakesh/Zookeeper/18_Aug/zookeeper-3.5.1-alpha/bin/../src/java/lib/ant-eclipse-1.0-jvm1.2.jar:/home/Rakesh/Zookeeper/18_Aug/zookeeper-3.5.1-alpha/bin/../conf:
2016-08-25 14:22:15,087 [myid:] - INFO [main:Environment@109] - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2016-08-25 14:22:15,087 [myid:] - INFO [main:Environment@109] - Client environment:java.io.tmpdir=/tmp
2016-08-25 14:22:15,087 [myid:] - INFO [main:Environment@109] - Client environment:java.compiler=
2016-08-25 14:22:15,087 [myid:] - INFO [main:Environment@109] - Client environment:os.name=Linux
2016-08-25 14:22:15,087 [myid:] - INFO [main:Environment@109] - Client environment:os.arch=amd64
2016-08-25 14:22:15,087 [myid:] - INFO [main:Environment@109] - Client environment:os.version=3.0.76-0.11-default
2016-08-25 14:22:15,087 [myid:] - INFO [main:Environment@109] - Client environment:user.name=root
2016-08-25 14:22:15,087 [myid:] - INFO [main:Environment@109] - Client environment:user.home=/root
2016-08-25 14:22:15,088 [myid:] - INFO [main:Environment@109] - Client environment:user.dir=/home/Rakesh/Zookeeper/18_Aug/zookeeper-3.5.1-alpha/bin
2016-08-25 14:22:15,088 [myid:] - INFO [main:Environment@109] - Client environment:os.memory.free=52MB
2016-08-25 14:22:15,090 [myid:] - INFO [main:Environment@109] - Client environment:os.memory.max=227MB
2016-08-25 14:22:15,090 [myid:] - INFO [main:Environment@109] - Client environment:os.memory.total=57MB
2016-08-25 14:22:15,095 [myid:] - INFO [main:ZooKeeper@716] - Initiating client connection, connectString=localhost:2181 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@272f15b0
Welcome to ZooKeeper!
2016-08-25 14:22:15,182 [myid:] - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread@1138] - Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
JLine support is enabled
[INFO] Unable to bind key for unsupported operation: backward-delete-word
[INFO] Unable to bind key for unsupported operation: backward-delete-word
[INFO] Unable to bind key for unsupported operation: down-history
[INFO] Unable to bind key for unsupported operation: up-history
[INFO] Unable to bind key for unsupported operation: up-history
[INFO] Unable to bind key for unsupported operation: down-history
[INFO] Unable to bind key for unsupported operation: up-history
[INFO] Unable to bind key for unsupported operation: down-history
[INFO] Unable to bind key for unsupported operation: up-history
[INFO] Unable to bind key for unsupported operation: down-history
[INFO] Unable to bind key for unsupported operation: up-history
[INFO] Unable to bind key for unsupported operation: down-history
[zk: localhost:2181(CONNECTING) 0] 2016-08-25 14:22:15,502 [myid:] - INFO [main-SendThread(localhost:2181):ClientCnxnSocketNetty$ZKClientPipelineFactory@363] - SSL handler added for channel: null
2016-08-25 14:22:15,537 [myid:] - INFO [New I/O worker #1:ClientCnxn$SendThread@980] - Socket connection established, initiating session, client: /127.0.0.1:57420, server: localhost/127.0.0.1:2181
2016-08-25 14:22:15,540 [myid:] - INFO [New I/O worker #1:ClientCnxnSocketNetty$1@146] - channel is connected: [id: 0xfc4fe483, /127.0.0.1:57420 => localhost/127.0.0.1:2181]
2016-08-25 14:22:15,673 [myid:] - INFO [New I/O worker #1:ClientCnxn$SendThread@1400] - Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x1019f8940e20001, negotiated timeout = 30000

WATCHER::

WatchedEvent state:SyncConnected type:None path:null

[zk: localhost:2181(CONNECTED) 0] removewatches /test -a
2016-08-25 14:24:14,420 [myid:] - ERROR [New I/O worker #1:ClientCnxn@725] - Failed to find watcher!
org.apache.zookeeper.KeeperException$NoWatcherException: KeeperErrorCode = No such watcher for /test
at org.apache.zookeeper.ZooKeeper$ZKWatchManager.containsWatcher(ZooKeeper.java:377)
at org.apache.zookeeper.ZooKeeper$ZKWatchManager.removeWatcher(ZooKeeper.java:252)
at org.apache.zookeeper.WatchDeregistration.unregister(WatchDeregistration.java:58)
at org.apache.zookeeper.ClientCnxn.finishPacket(ClientCnxn.java:712)
at org.apache.zookeeper.ClientCnxn.access$1500(ClientCnxn.java:97)
at org.apache.zookeeper.ClientCnxn$SendThread.readResponse(ClientCnxn.java:948)
at org.apache.zookeeper.ClientCnxnSocketNetty$ZKClientHandler.messageReceived(ClientCnxnSocketNetty.java:419)
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462)
at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443)
at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:109)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:90)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
KeeperErrorCode = No such watcher for /test
[zk: localhost:2181(CONNECTED) 1]
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
2 years, 38 weeks, 1 day ago 0|i32ron:
ZooKeeper ZOOKEEPER-2529

ZOOKEEPER-3170 Flaky Test: org.apache.zookeeper.test.NonRecoverableErrorTest.testZooKeeperServiceAvailableOnLeader

Sub-task Resolved Major Cannot Reproduce Andor Molnar Michael Han Michael Han 24/Aug/16 20:14   25/Oct/18 10:58 25/Oct/18 10:58 3.4.9, 3.5.3       0 2   This flaky is introduced by ZOOKEEPER-2247 recently.

{noformat}
Error Message

IOException is expected due to error injected to transaction log commit
Stacktrace

junit.framework.AssertionFailedError: IOException is expected due to error injected to transaction log commit
at org.apache.zookeeper.test.NonRecoverableErrorTest.testZooKeeperServiceAvailableOnLeader(NonRecoverableErrorTest.java:115)
at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:79)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.lang.Thread.run(Thread.java:745)
Standard Output

2016-08-24 19:20:17,000 [myid:] - INFO [main:JUnit4ZKTestRunner@47] - No test.method specified. using default methods.
2016-08-24 19:20:17,142 [myid:] - INFO [main:JUnit4ZKTestRunner@47] - No test.method specified. using default methods.
2016-08-24 19:20:17,215 [myid:] - INFO [main:ZKTestCase$1@55] - STARTING testZooKeeperServiceAvailableOnLeader
2016-08-24 19:20:17,222 [myid:] - INFO [Time-limited test:JUnit4ZKTestRunner$LoggedInvokeMethod@77] - RUNNING TEST METHOD testZooKeeperServiceAvailableOnLeader
2016-08-24 19:20:17,228 [myid:] - INFO [Time-limited test:PortAssignment@151] - Test process 1/8 using ports from 11221 - 13913.
2016-08-24 19:20:17,230 [myid:] - INFO [Time-limited test:PortAssignment@85] - Assigned port 11222 from range 11221 - 13913.
2016-08-24 19:20:17,231 [myid:] - INFO [Time-limited test:PortAssignment@85] - Assigned port 11223 from range 11221 - 13913.
2016-08-24 19:20:17,231 [myid:] - INFO [Time-limited test:PortAssignment@85] - Assigned port 11224 from range 11221 - 13913.
2016-08-24 19:20:17,232 [myid:] - INFO [Time-limited test:PortAssignment@85] - Assigned port 11225 from range 11221 - 13913.
2016-08-24 19:20:17,232 [myid:] - INFO [Time-limited test:PortAssignment@85] - Assigned port 11226 from range 11221 - 13913.
2016-08-24 19:20:17,232 [myid:] - INFO [Time-limited test:PortAssignment@85] - Assigned port 11227 from range 11221 - 13913.
2016-08-24 19:20:17,233 [myid:] - INFO [Time-limited test:PortAssignment@85] - Assigned port 11228 from range 11221 - 13913.
2016-08-24 19:20:17,233 [myid:] - INFO [Time-limited test:PortAssignment@85] - Assigned port 11229 from range 11221 - 13913.
2016-08-24 19:20:17,234 [myid:] - INFO [Time-limited test:PortAssignment@85] - Assigned port 11230 from range 11221 - 13913.
2016-08-24 19:20:17,256 [myid:] - INFO [Time-limited test:QuorumPeerTestBase$MainThread@131] - id = 0 tmpDir = /home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk/trunk/build/test/tmp/test7330020881446886387.junit.dir clientPort = 11222 adminServerPort = 8080
2016-08-24 19:20:17,262 [myid:] - INFO [Time-limited test:QuorumPeerTestBase$MainThread@131] - id = 1 tmpDir = /home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk/trunk/build/test/tmp/test7260737707555606353.junit.dir clientPort = 11225 adminServerPort = 8080
2016-08-24 19:20:17,267 [myid:] - INFO [Time-limited test:QuorumPeerTestBase$MainThread@131] - id = 2 tmpDir = /home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk/trunk/build/test/tmp/test1038022970424066351.junit.dir clientPort = 11228 adminServerPort = 8080
2016-08-24 19:20:17,268 [myid:] - INFO [Thread-1:QuorumPeerConfig@116] - Reading configuration from: /home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk/trunk/build/test/tmp/test7260737707555606353.junit.dir/zoo.cfg
2016-08-24 19:20:17,268 [myid:] - INFO [Thread-0:QuorumPeerConfig@116] - Reading configuration from: /home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk/trunk/build/test/tmp/test7330020881446886387.junit.dir/zoo.cfg
2016-08-24 19:20:17,269 [myid:] - INFO [Thread-2:QuorumPeerConfig@116] - Reading configuration from: /home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk/trunk/build/test/tmp/test1038022970424066351.junit.dir/zoo.cfg
2016-08-24 19:20:17,270 [myid:] - INFO [Thread-1:QuorumPeerConfig@318] - clientPortAddress is 0.0.0.0/0.0.0.0:11225
2016-08-24 19:20:17,270 [myid:] - INFO [Thread-0:QuorumPeerConfig@318] - clientPortAddress is 0.0.0.0/0.0.0.0:11222
2016-08-24 19:20:17,271 [myid:] - INFO [Thread-0:QuorumPeerConfig@322] - secureClientPort is not set
2016-08-24 19:20:17,270 [myid:] - INFO [Thread-1:QuorumPeerConfig@322] - secureClientPort is not set
2016-08-24 19:20:17,270 [myid:] - INFO [Thread-2:QuorumPeerConfig@318] - clientPortAddress is 0.0.0.0/0.0.0.0:11228
2016-08-24 19:20:17,271 [myid:] - INFO [Thread-2:QuorumPeerConfig@322] - secureClientPort is not set
2016-08-24 19:20:17,276 [myid:] - INFO [Time-limited test:FourLetterWordMain@85] - connecting to 127.0.0.1 11222
2016-08-24 19:20:17,280 [myid:] - INFO [Time-limited test:ClientBase@248] - server 127.0.0.1:11222 not up
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.client.FourLetterWordMain.send4LetterWord(FourLetterWordMain.java:99)
at org.apache.zookeeper.client.FourLetterWordMain.send4LetterWord(FourLetterWordMain.java:69)
at org.apache.zookeeper.test.ClientBase.waitForServerUp(ClientBase.java:241)
at org.apache.zookeeper.test.ClientBase.waitForServerUp(ClientBase.java:232)
at org.apache.zookeeper.test.NonRecoverableErrorTest.testZooKeeperServiceAvailableOnLeader(NonRecoverableErrorTest.java:77)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:79)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.lang.Thread.run(Thread.java:745)
2016-08-24 19:20:17,295 [myid:1] - INFO [Thread-1:DatadirCleanupManager@78] - autopurge.snapRetainCount set to 3
2016-08-24 19:20:17,295 [myid:2] - INFO [Thread-2:DatadirCleanupManager@78] - autopurge.snapRetainCount set to 3
2016-08-24 19:20:17,295 [myid:2] - INFO [Thread-2:DatadirCleanupManager@79] - autopurge.purgeInterval set to 0
2016-08-24 19:20:17,295 [myid:2] - INFO [Thread-2:DatadirCleanupManager@101] - Purge task is not scheduled.
2016-08-24 19:20:17,295 [myid:1] - INFO [Thread-1:DatadirCleanupManager@79] - autopurge.purgeInterval set to 0
2016-08-24 19:20:17,296 [myid:1] - INFO [Thread-1:DatadirCleanupManager@101] - Purge task is not scheduled.
2016-08-24 19:20:17,297 [myid:2] - INFO [Thread-2:ManagedUtil@46] - Log4j found with jmx enabled.
2016-08-24 19:20:17,298 [myid:1] - INFO [Thread-1:ManagedUtil@46] - Log4j found with jmx enabled.
2016-08-24 19:20:17,300 [myid:0] - INFO [Thread-0:DatadirCleanupManager@78] - autopurge.snapRetainCount set to 3
2016-08-24 19:20:17,300 [myid:0] - INFO [Thread-0:DatadirCleanupManager@79] - autopurge.purgeInterval set to 0
2016-08-24 19:20:17,300 [myid:0] - INFO [Thread-0:DatadirCleanupManager@101] - Purge task is not scheduled.
2016-08-24 19:20:17,300 [myid:0] - INFO [Thread-0:ManagedUtil@46] - Log4j found with jmx enabled.
2016-08-24 19:20:17,417 [myid:2] - ERROR [Thread-2:ManagedUtil@114] - Problems while registering log4j jmx beans!
javax.management.InstanceAlreadyExistsException: log4j:hiearchy=default
at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:437)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1898)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:966)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324)
at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
at org.apache.zookeeper.jmx.ManagedUtil.registerLog4jMBeans(ManagedUtil.java:75)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:131)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:120)
at org.apache.zookeeper.server.quorum.QuorumPeerTestBase$MainThread.run(QuorumPeerTestBase.java:245)
at java.lang.Thread.run(Thread.java:745)
2016-08-24 19:20:17,418 [myid:2] - WARN [Thread-2:QuorumPeerMain@133] - Unable to register log4j JMX control
javax.management.JMException: javax.management.InstanceAlreadyExistsException: log4j:hiearchy=default
at org.apache.zookeeper.jmx.ManagedUtil.registerLog4jMBeans(ManagedUtil.java:115)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:131)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:120)
at org.apache.zookeeper.server.quorum.QuorumPeerTestBase$MainThread.run(QuorumPeerTestBase.java:245)
at java.lang.Thread.run(Thread.java:745)
2016-08-24 19:20:17,425 [myid:2] - INFO [Thread-2:QuorumPeerMain@136] - Starting quorum peer
2016-08-24 19:20:17,443 [myid:2] - INFO [Thread-2:NIOServerCnxnFactory@673] - Configuring NIO connection handler with 10s sessionless connection timeout, 2 selector thread(s), 32 worker threads, and 64 kB direct buffers.
2016-08-24 19:20:17,444 [myid:0] - ERROR [Thread-0:HierarchyDynamicMBean@138] - Could not add loggerMBean for [root].
javax.management.InstanceAlreadyExistsException: log4j:logger=root
at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:437)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1898)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:966)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324)
at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
at org.apache.log4j.jmx.AbstractDynamicMBean.registerMBean(AbstractDynamicMBean.java:160)
at org.apache.log4j.jmx.HierarchyDynamicMBean.addLoggerMBean(HierarchyDynamicMBean.java:125)
at org.apache.log4j.jmx.HierarchyDynamicMBean.postRegister(HierarchyDynamicMBean.java:263)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.postRegister(DefaultMBeanServerInterceptor.java:1024)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:974)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324)
at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
at org.apache.zookeeper.jmx.ManagedUtil.registerLog4jMBeans(ManagedUtil.java:75)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:131)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:120)
at org.apache.zookeeper.server.quorum.QuorumPeerTestBase$MainThread.run(QuorumPeerTestBase.java:245)
at java.lang.Thread.run(Thread.java:745)
2016-08-24 19:20:17,445 [myid:0] - ERROR [Thread-0:ManagedUtil@114] - Problems while registering log4j jmx beans!
javax.management.InstanceAlreadyExistsException: log4j:hiearchy=default
at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:437)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1898)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:966)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324)
at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
at org.apache.zookeeper.jmx.ManagedUtil.registerLog4jMBeans(ManagedUtil.java:75)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:131)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:120)
at org.apache.zookeeper.server.quorum.QuorumPeerTestBase$MainThread.run(QuorumPeerTestBase.java:245)
at java.lang.Thread.run(Thread.java:745)
2016-08-24 19:20:17,446 [myid:0] - WARN [Thread-0:QuorumPeerMain@133] - Unable to register log4j JMX control
javax.management.JMException: javax.management.InstanceAlreadyExistsException: log4j:hiearchy=default
at org.apache.zookeeper.jmx.ManagedUtil.registerLog4jMBeans(ManagedUtil.java:115)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:131)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:120)
at org.apache.zookeeper.server.quorum.QuorumPeerTestBase$MainThread.run(QuorumPeerTestBase.java:245)
at java.lang.Thread.run(Thread.java:745)
2016-08-24 19:20:17,447 [myid:0] - INFO [Thread-0:QuorumPeerMain@136] - Starting quorum peer
2016-08-24 19:20:17,447 [myid:0] - INFO [Thread-0:NIOServerCnxnFactory@673] - Configuring NIO connection handler with 10s sessionless connection timeout, 2 selector thread(s), 32 worker threads, and 64 kB direct buffers.
2016-08-24 19:20:17,454 [myid:0] - INFO [Thread-0:NIOServerCnxnFactory@686] - binding to port /127.0.0.1:11222
2016-08-24 19:20:17,454 [myid:2] - INFO [Thread-2:NIOServerCnxnFactory@686] - binding to port /127.0.0.1:11228
2016-08-24 19:20:17,473 [myid:1] - INFO [Thread-1:QuorumPeerMain@136] - Starting quorum peer
2016-08-24 19:20:17,473 [myid:1] - INFO [Thread-1:NIOServerCnxnFactory@673] - Configuring NIO connection handler with 10s sessionless connection timeout, 2 selector thread(s), 32 worker threads, and 64 kB direct buffers.
2016-08-24 19:20:17,474 [myid:1] - INFO [Thread-1:NIOServerCnxnFactory@686] - binding to port /127.0.0.1:11225
2016-08-24 19:20:17,488 [myid:0] - INFO [Thread-0:QuorumPeer@1327] - Local sessions disabled
2016-08-24 19:20:17,488 [myid:2] - INFO [Thread-2:QuorumPeer@1327] - Local sessions disabled
2016-08-24 19:20:17,488 [myid:1] - INFO [Thread-1:QuorumPeer@1327] - Local sessions disabled
2016-08-24 19:20:17,488 [myid:2] - INFO [Thread-2:QuorumPeer@1338] - Local session upgrading disabled
2016-08-24 19:20:17,488 [myid:0] - INFO [Thread-0:QuorumPeer@1338] - Local session upgrading disabled
2016-08-24 19:20:17,488 [myid:0] - INFO [Thread-0:QuorumPeer@1305] - tickTime set to 4000
2016-08-24 19:20:17,488 [myid:2] - INFO [Thread-2:QuorumPeer@1305] - tickTime set to 4000
2016-08-24 19:20:17,489 [myid:2] - INFO [Thread-2:QuorumPeer@1349] - minSessionTimeout set to 8000
2016-08-24 19:20:17,489 [myid:2] - INFO [Thread-2:QuorumPeer@1360] - maxSessionTimeout set to 80000
2016-08-24 19:20:17,489 [myid:2] - INFO [Thread-2:QuorumPeer@1375] - initLimit set to 10
2016-08-24 19:20:17,488 [myid:1] - INFO [Thread-1:QuorumPeer@1338] - Local session upgrading disabled
2016-08-24 19:20:17,490 [myid:1] - INFO [Thread-1:QuorumPeer@1305] - tickTime set to 4000
2016-08-24 19:20:17,490 [myid:1] - INFO [Thread-1:QuorumPeer@1349] - minSessionTimeout set to 8000
2016-08-24 19:20:17,490 [myid:1] - INFO [Thread-1:QuorumPeer@1360] - maxSessionTimeout set to 80000
2016-08-24 19:20:17,490 [myid:1] - INFO [Thread-1:QuorumPeer@1375] - initLimit set to 10
2016-08-24 19:20:17,488 [myid:0] - INFO [Thread-0:QuorumPeer@1349] - minSessionTimeout set to 8000
2016-08-24 19:20:17,491 [myid:0] - INFO [Thread-0:QuorumPeer@1360] - maxSessionTimeout set to 80000
2016-08-24 19:20:17,491 [myid:0] - INFO [Thread-0:QuorumPeer@1375] - initLimit set to 10
2016-08-24 19:20:17,514 [myid:2] - INFO [Thread-2:QuorumPeer@776] - currentEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
2016-08-24 19:20:17,514 [myid:1] - INFO [Thread-1:QuorumPeer@776] - currentEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
2016-08-24 19:20:17,515 [myid:0] - INFO [Thread-0:QuorumPeer@776] - currentEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
2016-08-24 19:20:17,516 [myid:0] - INFO [Thread-0:QuorumPeer@791] - acceptedEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
2016-08-24 19:20:17,516 [myid:2] - INFO [Thread-2:QuorumPeer@791] - acceptedEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
2016-08-24 19:20:17,521 [myid:1] - INFO [Thread-1:QuorumPeer@791] - acceptedEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
2016-08-24 19:20:17,535 [myid:] - INFO [Time-limited test:FourLetterWordMain@85] - connecting to 127.0.0.1 11222
2016-08-24 19:20:17,539 [myid:0] - INFO [NIOServerCxnFactory.AcceptThread:/127.0.0.1:11222:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:41083
2016-08-24 19:20:17,554 [myid:0] - INFO [NIOWorkerThread-1:NIOServerCnxn@485] - Processing stat command from /127.0.0.1:41083
2016-08-24 19:20:17,568 [myid:0] - INFO [NIOWorkerThread-1:NIOServerCnxn@607] - Closed socket connection for client /127.0.0.1:41083 (no session established for client)
2016-08-24 19:20:17,595 [myid:0] - INFO [QuorumPeerListener:QuorumCnxManager$Listener@632] - My election bind port: /127.0.0.1:11224
2016-08-24 19:20:17,606 [myid:1] - INFO [QuorumPeerListener:QuorumCnxManager$Listener@632] - My election bind port: /127.0.0.1:11227
2016-08-24 19:20:17,607 [myid:0] - INFO [QuorumPeer[myid=0](plain=/127.0.0.1:11222)(secure=disabled):QuorumPeer@1033] - LOOKING
2016-08-24 19:20:17,611 [myid:1] - INFO [QuorumPeer[myid=1](plain=/127.0.0.1:11225)(secure=disabled):QuorumPeer@1033] - LOOKING
2016-08-24 19:20:17,624 [myid:0] - INFO [QuorumPeer[myid=0](plain=/127.0.0.1:11222)(secure=disabled):FastLeaderElection@894] - New election. My id = 0, proposed zxid=0x0
2016-08-24 19:20:17,626 [myid:1] - INFO [QuorumPeer[myid=1](plain=/127.0.0.1:11225)(secure=disabled):FastLeaderElection@894] - New election. My id = 1, proposed zxid=0x0
2016-08-24 19:20:17,634 [myid:2] - INFO [QuorumPeerListener:QuorumCnxManager$Listener@632] - My election bind port: /127.0.0.1:11230
2016-08-24 19:20:17,640 [myid:2] - INFO [QuorumPeer[myid=2](plain=/127.0.0.1:11228)(secure=disabled):QuorumPeer@1033] - LOOKING
2016-08-24 19:20:17,640 [myid:2] - INFO [QuorumPeer[myid=2](plain=/127.0.0.1:11228)(secure=disabled):FastLeaderElection@894] - New election. My id = 2, proposed zxid=0x0
2016-08-24 19:20:17,642 [myid:0] - INFO [/127.0.0.1:11224:QuorumCnxManager$Listener@638] - Received connection request /127.0.0.1:45368
2016-08-24 19:20:17,645 [myid:0] - INFO [WorkerReceiver[myid=0]:FastLeaderElection@688] - Notification: 2 (message format version), 0 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 0 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version)
2016-08-24 19:20:17,649 [myid:0] - INFO [/127.0.0.1:11224:QuorumCnxManager$Listener@638] - Received connection request /127.0.0.1:45369
2016-08-24 19:20:17,650 [myid:1] - INFO [/127.0.0.1:11227:QuorumCnxManager$Listener@638] - Received connection request /127.0.0.1:43290
2016-08-24 19:20:17,653 [myid:0] - INFO [WorkerReceiver[myid=0]:FastLeaderElection@688] - Notification: 2 (message format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version)
2016-08-24 19:20:17,654 [myid:0] - INFO [WorkerSender[myid=0]:QuorumCnxManager@276] - Have smaller server identifier, so dropping the connection: (1, 0)
2016-08-24 19:20:17,657 [myid:2] - INFO [/127.0.0.1:11230:QuorumCnxManager$Listener@638] - Received connection request /127.0.0.1:52198
2016-08-24 19:20:17,657 [myid:1] - INFO [WorkerSender[myid=1]:QuorumCnxManager@276] - Have smaller server identifier, so dropping the connection: (2, 1)
2016-08-24 19:20:17,659 [myid:0] - INFO [WorkerReceiver[myid=0]:FastLeaderElection@688] - Notification: 2 (message format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 0 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version)
2016-08-24 19:20:17,659 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@688] - Notification: 2 (message format version), 1 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version)
2016-08-24 19:20:17,659 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@688] - Notification: 2 (message format version), 0 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 0 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version)
2016-08-24 19:20:17,679 [myid:2] - WARN [SendWorker:1:QuorumCnxManager$SendWorker@841] - Send worker leaving thread id 1 my id = 2
2016-08-24 19:20:17,680 [myid:2] - WARN [RecvWorker:1:QuorumCnxManager$RecvWorker@918] - Interrupting SendWorker
2016-08-24 19:20:17,681 [myid:1] - INFO [/127.0.0.1:11227:QuorumCnxManager$Listener@638] - Received connection request /127.0.0.1:43291
2016-08-24 19:20:17,683 [myid:0] - INFO [/127.0.0.1:11224:QuorumCnxManager$Listener@638] - Received connection request /127.0.0.1:45377
2016-08-24 19:20:17,683 [myid:1] - WARN [SendWorker:0:QuorumCnxManager$SendWorker@832] - Interrupted while waiting for message on queue
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:982)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:63)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:820)
2016-08-24 19:20:17,684 [myid:1] - WARN [SendWorker:0:QuorumCnxManager$SendWorker@841] - Send worker leaving thread id 0 my id = 1
2016-08-24 19:20:17,686 [myid:1] - WARN [RecvWorker:0:QuorumCnxManager$RecvWorker@915] - Connection broken for id 0, my id = 1, error =
java.net.SocketException: Socket closed
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at java.net.SocketInputStream.read(SocketInputStream.java:210)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:900)
2016-08-24 19:20:17,686 [myid:1] - WARN [RecvWorker:0:QuorumCnxManager$RecvWorker@918] - Interrupting SendWorker
2016-08-24 19:20:17,687 [myid:0] - INFO [WorkerReceiver[myid=0]:FastLeaderElection@688] - Notification: 2 (message format version), 1 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version)
2016-08-24 19:20:17,687 [myid:0] - WARN [RecvWorker:1:QuorumCnxManager$RecvWorker@915] - Connection broken for id 1, my id = 0, error =
java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:392)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:900)
2016-08-24 19:20:17,699 [myid:0] - WARN [RecvWorker:1:QuorumCnxManager$RecvWorker@918] - Interrupting SendWorker
2016-08-24 19:20:17,696 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@688] - Notification: 2 (message format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version)
2016-08-24 19:20:17,701 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@688] - Notification: 2 (message format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 0 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version)
2016-08-24 19:20:17,702 [myid:0] - WARN [SendWorker:1:QuorumCnxManager$SendWorker@837] - Exception when using channel: for id 1 my id = 0 error = java.net.SocketException: Broken pipe
2016-08-24 19:20:17,688 [myid:1] - WARN [SendWorker:2:QuorumCnxManager$SendWorker@837] - Exception when using channel: for id 2 my id = 1 error = java.net.SocketException: Broken pipe
2016-08-24 19:20:17,701 [myid:1] - WARN [RecvWorker:2:QuorumCnxManager$RecvWorker@915] - Connection broken for id 2, my id = 1, error =
java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:392)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:900)
2016-08-24 19:20:17,703 [myid:1] - WARN [RecvWorker:2:QuorumCnxManager$RecvWorker@918] - Interrupting SendWorker
2016-08-24 19:20:17,689 [myid:1] - INFO [/127.0.0.1:11227:QuorumCnxManager$Listener@638] - Received connection request /127.0.0.1:43293
2016-08-24 19:20:17,708 [myid:1] - WARN [SendWorker:2:QuorumCnxManager$SendWorker@841] - Send worker leaving thread id 2 my id = 1
2016-08-24 19:20:17,708 [myid:0] - WARN [SendWorker:1:QuorumCnxManager$SendWorker@841] - Send worker leaving thread id 1 my id = 0
2016-08-24 19:20:17,710 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@688] - Notification: 2 (message format version), 1 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version)
2016-08-24 19:20:17,710 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@688] - Notification: 2 (message format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version)
2016-08-24 19:20:17,720 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@688] - Notification: 2 (message format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version)
2016-08-24 19:20:17,722 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@688] - Notification: 2 (message format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 0 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version)
2016-08-24 19:20:17,728 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@688] - Notification: 2 (message format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version)
2016-08-24 19:20:17,728 [myid:0] - INFO [WorkerReceiver[myid=0]:FastLeaderElection@688] - Notification: 2 (message format version), 1 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version)
2016-08-24 19:20:17,729 [myid:0] - INFO [WorkerReceiver[myid=0]:FastLeaderElection@688] - Notification: 2 (message format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version)
2016-08-24 19:20:17,819 [myid:] - INFO [Time-limited test:FourLetterWordMain@85] - connecting to 127.0.0.1 11222
2016-08-24 19:20:17,820 [myid:0] - INFO [NIOServerCxnFactory.AcceptThread:/127.0.0.1:11222:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:41114
2016-08-24 19:20:17,822 [myid:0] - INFO [NIOWorkerThread-2:NIOServerCnxn@485] - Processing stat command from /127.0.0.1:41114
2016-08-24 19:20:17,822 [myid:0] - INFO [NIOWorkerThread-2:NIOServerCnxn@607] - Closed socket connection for client /127.0.0.1:41114 (no session established for client)
2016-08-24 19:20:17,923 [myid:1] - INFO [QuorumPeer[myid=1](plain=/127.0.0.1:11225)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id1,name1=replica.1,name2=LeaderElection]
2016-08-24 19:20:17,923 [myid:1] - INFO [QuorumPeer[myid=1](plain=/127.0.0.1:11225)(secure=disabled):QuorumPeer@1109] - FOLLOWING
2016-08-24 19:20:17,928 [myid:2] - INFO [QuorumPeer[myid=2](plain=/127.0.0.1:11228)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id2,name1=replica.2,name2=LeaderElection]
2016-08-24 19:20:17,929 [myid:2] - INFO [QuorumPeer[myid=2](plain=/127.0.0.1:11228)(secure=disabled):QuorumPeer@1121] - LEADING
2016-08-24 19:20:17,929 [myid:0] - INFO [QuorumPeer[myid=0](plain=/127.0.0.1:11222)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id0,name1=replica.0,name2=LeaderElection]
2016-08-24 19:20:17,930 [myid:0] - INFO [QuorumPeer[myid=0](plain=/127.0.0.1:11222)(secure=disabled):QuorumPeer@1109] - FOLLOWING
2016-08-24 19:20:17,932 [myid:1] - INFO [QuorumPeer[myid=1](plain=/127.0.0.1:11225)(secure=disabled):Learner@88] - TCP NoDelay set to: true
2016-08-24 19:20:17,937 [myid:2] - INFO [QuorumPeer[myid=2](plain=/127.0.0.1:11228)(secure=disabled):Leader@63] - TCP NoDelay set to: true
2016-08-24 19:20:17,937 [myid:2] - INFO [QuorumPeer[myid=2](plain=/127.0.0.1:11228)(secure=disabled):Leader@83] - zookeeper.leader.maxConcurrentSnapshots = 10
2016-08-24 19:20:17,938 [myid:2] - INFO [QuorumPeer[myid=2](plain=/127.0.0.1:11228)(secure=disabled):Leader@85] - zookeeper.leader.maxConcurrentSnapshotTimeout = 5
2016-08-24 19:20:17,942 [myid:0] - INFO [QuorumPeer[myid=0](plain=/127.0.0.1:11222)(secure=disabled):Environment@109] - Server environment:zookeeper.version=3.6.0-SNAPSHOT-1757568, built on 08/24/2016 19:17 GMT
2016-08-24 19:20:17,942 [myid:0] - INFO [QuorumPeer[myid=0](plain=/127.0.0.1:11222)(secure=disabled):Environment@109] - Server environment:host.name=asf907.gq1.ygridcore.net
2016-08-24 19:20:17,942 [myid:0] - INFO [QuorumPeer[myid=0](plain=/127.0.0.1:11222)(secure=disabled):Environment@109] - Server environment:java.version=1.7.0_80
2016-08-24 19:20:17,943 [myid:0] - INFO [QuorumPeer[myid=0](plain=/127.0.0.1:11222)(secure=disabled):Environment@109] - Server environment:java.vendor=Oracle Corporation
2016-08-24 19:20:17,943 [myid:0] - INFO [QuorumPeer[myid=0](plain=/127.0.0.1:11222)(secure=disabled):Environment@109] - Server environment:java.home=/usr/local/asfpackages/java/jdk1.7.0_80/jre
2016-08-24 19:20:17,943 [myid:0] - INFO [QuorumPeer[myid=0](plain=/127.0.0.1:11222)(secure=disabled):Environment@109] - Server environment:java.class.path=/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk/trunk/build/test/classes:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk/trunk/build/test/lib/antlr-2.7.7.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk/trunk/build/test/lib/antlr4-runtime-4.5.1-1.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk/trunk/build/test/lib/checkstyle-6.13.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk/trunk/build/test/lib/commons-beanutils-1.9.2.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk/trunk/build/test/lib/commons-cli-1.3.1.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk/trunk/build/test/lib/commons-collections-3.2.2.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk/trunk/build/test/lib/commons-lang3-3.4.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk/trunk/build/test/lib/commons-logging-1.1.1.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk/trunk/build/test/lib/guava-18.0.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk/trunk/build/test/lib/hamcrest-core-1.3.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk/trunk/build/test/lib/junit-4.12.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk/trunk/build/test/lib/mockito-all-1.8.2.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk/trunk/build/classes:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk/trunk/src/java/lib/ivy-2.4.0.jar:/home/jenkins/tools/ant/latest/lib/ant.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk/trunk/build/lib/commons-cli-1.2.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk/trunk/build/lib/jackson-core-asl-1.9.11.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk/trunk/build/lib/jackson-mapper-asl-1.9.11.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk/trunk/build/lib/javacc.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk/trunk/build/lib/jetty-6.1.26.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk/trunk/build/lib/jetty-util-6.1.26.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk/trunk/build/lib/jline-2.11.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk/trunk/build/lib/log4j-1.2.17.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk/trunk/build/lib/netty-3.10.5.Final.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk/trunk/build/lib/servlet-api-2.5-20081211.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk/trunk/build/lib/slf4j-api-1.7.5.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk/trunk/build/lib/slf4j-log4j12-1.7.5.jar:/usr/local/asfpackages/ant/apache-ant-1.9.7/lib/ant-launcher.jar:/home/jenkins/tools/ant/latest/lib/ant-junit.jar:/home/jenkins/tools/ant/latest/lib/ant-junit4.jar
2016-08-24 19:20:17,943 [myid:0] - INFO [QuorumPeer[myid=0](plain=/127.0.0.1:11222)(secure=disabled):Environment@109] - Server environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2016-08-24 19:20:17,943 [myid:0] - INFO [QuorumPeer[myid=0](plain=/127.0.0.1:11222)(secure=disabled):Environment@109] - Server environment:java.io.tmpdir=/tmp
2016-08-24 19:20:17,943 [myid:0] - INFO [QuorumPeer[myid=0](plain=/127.0.0.1:11222)(secure=disabled):Environment@109] - Server environment:java.compiler=<NA>
2016-08-24 19:20:17,943 [myid:0] - INFO [QuorumPeer[myid=0](plain=/127.0.0.1:11222)(secure=disabled):Environment@109] - Server environment:os.name=Linux
2016-08-24 19:20:17,943 [myid:0] - INFO [QuorumPeer[myid=0](plain=/127.0.0.1:11222)(secure=disabled):Environment@109] - Server environment:os.arch=amd64
2016-08-24 19:20:17,944 [myid:0] - INFO [QuorumPeer[myid=0](plain=/127.0.0.1:11222)(secure=disabled):Environment@109] - Server environment:os.version=3.13.0-36-lowlatency
2016-08-24 19:20:17,944 [myid:0] - INFO [QuorumPeer[myid=0](plain=/127.0.0.1:11222)(secure=disabled):Environment@109] - Server environment:user.name=jenkins
2016-08-24 19:20:17,944 [myid:0] - INFO [QuorumPeer[myid=0](plain=/127.0.0.1:11222)(secure=disabled):Environment@109] - Server environment:user.home=/home/jenkins
2016-08-24 19:20:17,944 [myid:0] - INFO [QuorumPeer[myid=0](plain=/127.0.0.1:11222)(secure=disabled):Environment@109] - Server environment:user.dir=/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk/trunk/build/test
2016-08-24 19:20:17,944 [myid:0] - INFO [QuorumPeer[myid=0](plain=/127.0.0.1:11222)(secure=disabled):Environment@109] - Server environment:os.memory.free=369MB
2016-08-24 19:20:17,944 [myid:0] - INFO [QuorumPeer[myid=0](plain=/127.0.0.1:11222)(secure=disabled):Environment@109] - Server environment:os.memory.max=491MB
2016-08-24 19:20:17,944 [myid:0] - INFO [QuorumPeer[myid=0](plain=/127.0.0.1:11222)(secure=disabled):Environment@109] - Server environment:os.memory.total=491MB
2016-08-24 19:20:17,946 [myid:2] - INFO [QuorumPeer[myid=2](plain=/127.0.0.1:11228)(secure=disabled):ZooKeeperServer@889] - minSessionTimeout set to 8000
2016-08-24 19:20:17,946 [myid:1] - INFO [QuorumPeer[myid=1](plain=/127.0.0.1:11225)(secure=disabled):ZooKeeperServer@889] - minSessionTimeout set to 8000
2016-08-24 19:20:17,946 [myid:0] - INFO [QuorumPeer[myid=0](plain=/127.0.0.1:11222)(secure=disabled):ZooKeeperServer@889] - minSessionTimeout set to 8000
2016-08-24 19:20:17,946 [myid:1] - INFO [QuorumPeer[myid=1](plain=/127.0.0.1:11225)(secure=disabled):ZooKeeperServer@898] - maxSessionTimeout set to 80000
2016-08-24 19:20:17,948 [myid:1] - INFO [QuorumPeer[myid=1](plain=/127.0.0.1:11225)(secure=disabled):ZooKeeperServer@159] - Created server with tickTime 4000 minSessionTimeout 8000 maxSessionTimeout 80000 datadir /home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk/trunk/build/test/tmp/test7260737707555606353.junit.dir/data/version-2 snapdir /home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk/trunk/build/test/tmp/test7260737707555606353.junit.dir/data/version-2
2016-08-24 19:20:17,948 [myid:1] - INFO [QuorumPeer[myid=1](plain=/127.0.0.1:11225)(secure=disabled):Follower@66] - FOLLOWING - LEADER ELECTION TOOK - 25 MS
2016-08-24 19:20:17,946 [myid:2] - INFO [QuorumPeer[myid=2](plain=/127.0.0.1:11228)(secure=disabled):ZooKeeperServer@898] - maxSessionTimeout set to 80000
2016-08-24 19:20:17,947 [myid:0] - INFO [QuorumPeer[myid=0](plain=/127.0.0.1:11222)(secure=disabled):ZooKeeperServer@898] - maxSessionTimeout set to 80000
2016-08-24 19:20:17,953 [myid:2] - INFO [QuorumPeer[myid=2](plain=/127.0.0.1:11228)(secure=disabled):ZooKeeperServer@159] - Created server with tickTime 4000 minSessionTimeout 8000 maxSessionTimeout 80000 datadir /home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk/trunk/build/test/tmp/test1038022970424066351.junit.dir/data/version-2 snapdir /home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk/trunk/build/test/tmp/test1038022970424066351.junit.dir/data/version-2
2016-08-24 19:20:17,953 [myid:0] - INFO [QuorumPeer[myid=0](plain=/127.0.0.1:11222)(secure=disabled):ZooKeeperServer@159] - Created server with tickTime 4000 minSessionTimeout 8000 maxSessionTimeout 80000 datadir /home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk/trunk/build/test/tmp/test7330020881446886387.junit.dir/data/version-2 snapdir /home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk/trunk/build/test/tmp/test7330020881446886387.junit.dir/data/version-2
2016-08-24 19:20:17,953 [myid:0] - INFO [QuorumPeer[myid=0](plain=/127.0.0.1:11222)(secure=disabled):Follower@66] - FOLLOWING - LEADER ELECTION TOOK - 23 MS
2016-08-24 19:20:17,953 [myid:1] - WARN [QuorumPeer[myid=1](plain=/127.0.0.1:11225)(secure=disabled):Learner@273] - Unexpected exception, tries=0, remaining init limit=40000, connecting to /127.0.0.1:11229
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.server.quorum.Learner.sockConnect(Learner.java:227)
at org.apache.zookeeper.server.quorum.Learner.connectToLeader(Learner.java:256)
at org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:74)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1111)
2016-08-24 19:20:17,954 [myid:0] - WARN [QuorumPeer[myid=0](plain=/127.0.0.1:11222)(secure=disabled):Learner@273] - Unexpected exception, tries=0, remaining init limit=40000, connecting to /127.0.0.1:11229
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.server.quorum.Learner.sockConnect(Learner.java:227)
at org.apache.zookeeper.server.quorum.Learner.connectToLeader(Learner.java:256)
at org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:74)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1111)
2016-08-24 19:20:17,956 [myid:2] - INFO [QuorumPeer[myid=2](plain=/127.0.0.1:11228)(secure=disabled):Leader@412] - LEADING - LEADER ELECTION TOOK - 27 MS
2016-08-24 19:20:17,960 [myid:2] - INFO [QuorumPeer[myid=2](plain=/127.0.0.1:11228)(secure=disabled):FileTxnSnapLog@298] - Snapshotting: 0x0 to /home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk/trunk/build/test/tmp/test1038022970424066351.junit.dir/data/version-2/snapshot.0
2016-08-24 19:20:18,073 [myid:] - INFO [Time-limited test:FourLetterWordMain@85] - connecting to 127.0.0.1 11222
2016-08-24 19:20:18,075 [myid:0] - INFO [NIOServerCxnFactory.AcceptThread:/127.0.0.1:11222:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:41147
2016-08-24 19:20:18,080 [myid:0] - INFO [NIOWorkerThread-3:NIOServerCnxn@485] - Processing stat command from /127.0.0.1:41147
2016-08-24 19:20:18,080 [myid:0] - INFO [NIOWorkerThread-3:NIOServerCnxn@607] - Closed socket connection for client /127.0.0.1:41147 (no session established for client)
2016-08-24 19:20:18,332 [myid:] - INFO [Time-limited test:FourLetterWordMain@85] - connecting to 127.0.0.1 11222
2016-08-24 19:20:18,333 [myid:0] - INFO [NIOServerCxnFactory.AcceptThread:/127.0.0.1:11222:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:41177
2016-08-24 19:20:18,334 [myid:0] - INFO [NIOWorkerThread-4:NIOServerCnxn@485] - Processing stat command from /127.0.0.1:41177
2016-08-24 19:20:18,335 [myid:0] - INFO [NIOWorkerThread-4:NIOServerCnxn@607] - Closed socket connection for client /127.0.0.1:41177 (no session established for client)
2016-08-24 19:20:18,585 [myid:] - INFO [Time-limited test:FourLetterWordMain@85] - connecting to 127.0.0.1 11222
2016-08-24 19:20:18,586 [myid:0] - INFO [NIOServerCxnFactory.AcceptThread:/127.0.0.1:11222:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:41205
2016-08-24 19:20:18,587 [myid:0] - INFO [NIOWorkerThread-5:NIOServerCnxn@485] - Processing stat command from /127.0.0.1:41205
2016-08-24 19:20:18,588 [myid:0] - INFO [NIOWorkerThread-5:NIOServerCnxn@607] - Closed socket connection for client /127.0.0.1:41205 (no session established for client)
2016-08-24 19:20:18,838 [myid:] - INFO [Time-limited test:FourLetterWordMain@85] - connecting to 127.0.0.1 11222
2016-08-24 19:20:18,840 [myid:0] - INFO [NIOServerCxnFactory.AcceptThread:/127.0.0.1:11222:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:41223
2016-08-24 19:20:18,865 [myid:0] - INFO [NIOWorkerThread-6:NIOServerCnxn@485] - Processing stat command from /127.0.0.1:41223
2016-08-24 19:20:18,865 [myid:0] - INFO [NIOWorkerThread-6:NIOServerCnxn@607] - Closed socket connection for client /127.0.0.1:41223 (no session established for client)
2016-08-24 19:20:18,965 [myid:2] - INFO [LearnerHandler-/127.0.0.1:40597:LearnerHandler@382] - Follower sid: 1 : info : 127.0.0.1:11226:11227:participant;127.0.0.1:11225
2016-08-24 19:20:18,965 [myid:2] - INFO [LearnerHandler-/127.0.0.1:40598:LearnerHandler@382] - Follower sid: 0 : info : 127.0.0.1:11223:11224:participant;127.0.0.1:11222
2016-08-24 19:20:18,972 [myid:2] - INFO [LearnerHandler-/127.0.0.1:40598:LearnerHandler@683] - Synchronizing with Follower sid: 0 maxCommittedLog=0x0 minCommittedLog=0x0 lastProcessedZxid=0x0 peerLastZxid=0x0
2016-08-24 19:20:18,972 [myid:2] - INFO [LearnerHandler-/127.0.0.1:40597:LearnerHandler@683] - Synchronizing with Follower sid: 1 maxCommittedLog=0x0 minCommittedLog=0x0 lastProcessedZxid=0x0 peerLastZxid=0x0
2016-08-24 19:20:18,972 [myid:2] - INFO [LearnerHandler-/127.0.0.1:40598:LearnerHandler@727] - Sending DIFF zxid=0x0 for peer sid: 0
2016-08-24 19:20:18,972 [myid:2] - INFO [LearnerHandler-/127.0.0.1:40597:LearnerHandler@727] - Sending DIFF zxid=0x0 for peer sid: 1
2016-08-24 19:20:18,974 [myid:0] - INFO [QuorumPeer[myid=0](plain=/127.0.0.1:11222)(secure=disabled):Learner@366] - Getting a diff from the leader 0x0
2016-08-24 19:20:18,974 [myid:1] - INFO [QuorumPeer[myid=1](plain=/127.0.0.1:11225)(secure=disabled):Learner@366] - Getting a diff from the leader 0x0
2016-08-24 19:20:18,978 [myid:0] - INFO [QuorumPeer[myid=0](plain=/127.0.0.1:11222)(secure=disabled):Learner@509] - Learner received NEWLEADER message
2016-08-24 19:20:18,978 [myid:1] - INFO [QuorumPeer[myid=1](plain=/127.0.0.1:11225)(secure=disabled):Learner@509] - Learner received NEWLEADER message
2016-08-24 19:20:18,980 [myid:0] - INFO [QuorumPeer[myid=0](plain=/127.0.0.1:11222)(secure=disabled):FileTxnSnapLog@298] - Snapshotting: 0x0 to /home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk/trunk/build/test/tmp/test7330020881446886387.junit.dir/data/version-2/snapshot.0
2016-08-24 19:20:18,981 [myid:1] - INFO [QuorumPeer[myid=1](plain=/127.0.0.1:11225)(secure=disabled):FileTxnSnapLog@298] - Snapshotting: 0x0 to /home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk/trunk/build/test/tmp/test7260737707555606353.junit.dir/data/version-2/snapshot.0
2016-08-24 19:20:18,982 [myid:2] - INFO [QuorumPeer[myid=2](plain=/127.0.0.1:11228)(secure=disabled):Leader@1255] - Have quorum of supporters, sids: [ [0, 2],[0, 2] ]; starting up and setting last processed zxid: 0x100000000
2016-08-24 19:20:19,012 [myid:2] - INFO [QuorumPeer[myid=2](plain=/127.0.0.1:11228)(secure=disabled):CommitProcessor@318] - Configuring CommitProcessor with 16 worker threads.
2016-08-24 19:20:19,056 [myid:2] - INFO [QuorumPeer[myid=2](plain=/127.0.0.1:11228)(secure=disabled):ContainerManager@64] - Using checkIntervalMs=60000 maxPerMinute=10000
2016-08-24 19:20:19,062 [myid:1] - INFO [QuorumPeer[myid=1](plain=/127.0.0.1:11225)(secure=disabled):Learner@493] - Learner received UPTODATE message
2016-08-24 19:20:19,062 [myid:0] - INFO [QuorumPeer[myid=0](plain=/127.0.0.1:11222)(secure=disabled):Learner@493] - Learner received UPTODATE message
2016-08-24 19:20:19,067 [myid:0] - INFO [QuorumPeer[myid=0](plain=/127.0.0.1:11222)(secure=disabled):CommitProcessor@318] - Configuring CommitProcessor with 16 worker threads.
2016-08-24 19:20:19,080 [myid:1] - INFO [QuorumPeer[myid=1](plain=/127.0.0.1:11225)(secure=disabled):CommitProcessor@318] - Configuring CommitProcessor with 16 worker threads.
2016-08-24 19:20:19,116 [myid:] - INFO [Time-limited test:FourLetterWordMain@85] - connecting to 127.0.0.1 11222
2016-08-24 19:20:19,117 [myid:0] - INFO [NIOServerCxnFactory.AcceptThread:/127.0.0.1:11222:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:41246
2016-08-24 19:20:19,120 [myid:0] - INFO [NIOWorkerThread-7:NIOServerCnxn@485] - Processing stat command from /127.0.0.1:41246
2016-08-24 19:20:19,122 [myid:0] - INFO [NIOWorkerThread-7:StatCommand@49] - Stat command output
2016-08-24 19:20:19,123 [myid:0] - INFO [NIOWorkerThread-7:NIOServerCnxn@607] - Closed socket connection for client /127.0.0.1:41246 (no session established for client)
2016-08-24 19:20:19,123 [myid:] - INFO [Time-limited test:FourLetterWordMain@85] - connecting to 127.0.0.1 11225
2016-08-24 19:20:19,124 [myid:1] - INFO [NIOServerCxnFactory.AcceptThread:/127.0.0.1:11225:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:55153
2016-08-24 19:20:19,125 [myid:1] - INFO [NIOWorkerThread-1:NIOServerCnxn@485] - Processing stat command from /127.0.0.1:55153
2016-08-24 19:20:19,125 [myid:1] - INFO [NIOWorkerThread-1:StatCommand@49] - Stat command output
2016-08-24 19:20:19,126 [myid:1] - INFO [NIOWorkerThread-1:NIOServerCnxn@607] - Closed socket connection for client /127.0.0.1:55153 (no session established for client)
2016-08-24 19:20:19,126 [myid:] - INFO [Time-limited test:FourLetterWordMain@85] - connecting to 127.0.0.1 11228
2016-08-24 19:20:19,127 [myid:2] - INFO [NIOServerCxnFactory.AcceptThread:/127.0.0.1:11228:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:59066
2016-08-24 19:20:19,130 [myid:2] - INFO [NIOWorkerThread-1:NIOServerCnxn@485] - Processing stat command from /127.0.0.1:59066
2016-08-24 19:20:19,131 [myid:2] - INFO [NIOWorkerThread-1:StatCommand@49] - Stat command output
2016-08-24 19:20:19,131 [myid:2] - INFO [NIOWorkerThread-1:NIOServerCnxn@607] - Closed socket connection for client /127.0.0.1:59066 (no session established for client)
2016-08-24 19:20:19,138 [myid:] - INFO [Time-limited test:Environment@109] - Client environment:zookeeper.version=3.6.0-SNAPSHOT-1757568, built on 08/24/2016 19:17 GMT
2016-08-24 19:20:19,138 [myid:] - INFO [Time-limited test:Environment@109] - Client environment:host.name=asf907.gq1.ygridcore.net
2016-08-24 19:20:19,138 [myid:] - INFO [Time-limited test:Environment@109] - Client environment:java.version=1.7.0_80
2016-08-24 19:20:19,139 [myid:] - INFO [Time-limited test:Environment@109] - Client environment:java.vendor=Oracle Corporation
2016-08-24 19:20:19,139 [myid:] - INFO [Time-limited test:Environment@109] - Client environment:java.home=/usr/local/asfpackages/java/jdk1.7.0_80/jre
2016-08-24 19:20:19,139 [myid:] - INFO [Time-limited test:Environment@109] - Client environment:java.class.path=/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk/trunk/build/test/classes:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk/trunk/build/test/lib/antlr-2.7.7.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk/trunk/build/test/lib/antlr4-runtime-4.5.1-1.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk/trunk/build/test/lib/checkstyle-6.13.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk/trunk/build/test/lib/commons-beanutils-1.9.2.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk/trunk/build/test/lib/commons-cli-1.3.1.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk/trunk/build/test/lib/commons-collections-3.2.2.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk/trunk/build/test/lib/commons-lang3-3.4.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk/trunk/build/test/lib/commons-logging-1.1.1.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk/trunk/build/test/lib/guava-18.0.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk/trunk/build/test/lib/hamcrest-core-1.3.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk/trunk/build/test/lib/junit-4.12.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk/trunk/build/test/lib/mockito-all-1.8.2.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk/trunk/build/classes:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk/trunk/src/java/lib/ivy-2.4.0.jar:/home/jenkins/tools/ant/latest/lib/ant.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk/trunk/build/lib/commons-cli-1.2.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk/trunk/build/lib/jackson-core-asl-1.9.11.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk/trunk/build/lib/jackson-mapper-asl-1.9.11.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk/trunk/build/lib/javacc.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk/trunk/build/lib/jetty-6.1.26.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk/trunk/build/lib/jetty-util-6.1.26.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk/trunk/build/lib/jline-2.11.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk/trunk/build/lib/log4j-1.2.17.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk/trunk/build/lib/netty-3.10.5.Final.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk/trunk/build/lib/servlet-api-2.5-20081211.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk/trunk/build/lib/slf4j-api-1.7.5.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk/trunk/build/lib/slf4j-log4j12-1.7.5.jar:/usr/local/asfpackages/ant/apache-ant-1.9.7/lib/ant-launcher.jar:/home/jenkins/tools/ant/latest/lib/ant-junit.jar:/home/jenkins/tools/ant/latest/lib/ant-junit4.jar
2016-08-24 19:20:19,139 [myid:] - INFO [Time-limited test:Environment@109] - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2016-08-24 19:20:19,140 [myid:] - INFO [Time-limited test:Environment@109] - Client environment:java.io.tmpdir=/tmp
2016-08-24 19:20:19,140 [myid:] - INFO [Time-limited test:Environment@109] - Client environment:java.compiler=<NA>
2016-08-24 19:20:19,140 [myid:] - INFO [Time-limited test:Environment@109] - Client environment:os.name=Linux
2016-08-24 19:20:19,140 [myid:] - INFO [Time-limited test:Environment@109] - Client environment:os.arch=amd64
2016-08-24 19:20:19,141 [myid:] - INFO [Time-limited test:Environment@109] - Client environment:os.version=3.13.0-36-lowlatency
2016-08-24 19:20:19,141 [myid:] - INFO [Time-limited test:Environment@109] - Client environment:user.name=jenkins
2016-08-24 19:20:19,141 [myid:] - INFO [Time-limited test:Environment@109] - Client environment:user.home=/home/jenkins
2016-08-24 19:20:19,141 [myid:] - INFO [Time-limited test:Environment@109] - Client environment:user.dir=/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk/trunk/build/test
2016-08-24 19:20:19,141 [myid:] - INFO [Time-limited test:Environment@109] - Client environment:os.memory.free=462MB
2016-08-24 19:20:19,142 [myid:] - INFO [Time-limited test:Environment@109] - Client environment:os.memory.max=491MB
2016-08-24 19:20:19,142 [myid:] - INFO [Time-limited test:Environment@109] - Client environment:os.memory.total=491MB
2016-08-24 19:20:19,145 [myid:] - INFO [Time-limited test:ZooKeeper@855] - Initiating client connection, connectString=127.0.0.1:11222 sessionTimeout=30000 watcher=org.apache.zookeeper.test.ClientBase$CountdownWatcher@2b6c7d59
2016-08-24 19:20:19,169 [myid:127.0.0.1:11222] - INFO [Time-limited test-SendThread(127.0.0.1:11222):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:11222. Will not attempt to authenticate using SASL (unknown error)
2016-08-24 19:20:19,172 [myid:0] - INFO [NIOServerCxnFactory.AcceptThread:/127.0.0.1:11222:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:41255
2016-08-24 19:20:19,177 [myid:127.0.0.1:11222] - INFO [Time-limited test-SendThread(127.0.0.1:11222):ClientCnxn$SendThread@948] - Socket connection established, initiating session, client: /127.0.0.1:41255, server: 127.0.0.1/127.0.0.1:11222
2016-08-24 19:20:19,201 [myid:0] - INFO [NIOWorkerThread-8:ZooKeeperServer@995] - Client attempting to establish new session at /127.0.0.1:41255
2016-08-24 19:20:19,206 [myid:2] - INFO [SyncThread:2:FileTxnLog@204] - Creating new log file: log.100000001
2016-08-24 19:20:19,206 [myid:0] - WARN [QuorumPeer[myid=0](plain=/127.0.0.1:11222)(secure=disabled):Follower@122] - Got zxid 0x100000001 expected 0x1
2016-08-24 19:20:19,206 [myid:1] - WARN [QuorumPeer[myid=1](plain=/127.0.0.1:11225)(secure=disabled):Follower@122] - Got zxid 0x100000001 expected 0x1
2016-08-24 19:20:19,207 [myid:0] - INFO [SyncThread:0:FileTxnLog@204] - Creating new log file: log.100000001
2016-08-24 19:20:19,207 [myid:1] - INFO [SyncThread:1:FileTxnLog@204] - Creating new log file: log.100000001
2016-08-24 19:20:19,229 [myid:0] - INFO [CommitProcWorkThread-1:ZooKeeperServer@709] - Established session 0x2162abf710000 with negotiated timeout 30000 for client /127.0.0.1:41255
2016-08-24 19:20:19,231 [myid:127.0.0.1:11222] - INFO [Time-limited test-SendThread(127.0.0.1:11222):ClientCnxn$SendThread@1381] - Session establishment complete on server 127.0.0.1/127.0.0.1:11222, sessionid = 0x2162abf710000, negotiated timeout = 30000
2016-08-24 19:20:19,251 [myid:2] - INFO [SyncThread:2:FileTxnLog@204] - Creating new log file: log.100000003
2016-08-24 19:20:19,251 [myid:2] - ERROR [SyncThread:2:ZooKeeperCriticalThread@48] - Severe unrecoverable error, from thread : SyncThread:2
java.io.IOException: Input/output error
at org.apache.zookeeper.test.NonRecoverableErrorTest$1.commit(NonRecoverableErrorTest.java:101)
at org.apache.zookeeper.server.ZKDatabase.commit(ZKDatabase.java:557)
at org.apache.zookeeper.server.SyncRequestProcessor.flush(SyncRequestProcessor.java:178)
at org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:113)
2016-08-24 19:20:19,251 [myid:2] - INFO [SyncThread:2:ZooKeeperServerListenerImpl@42] - Thread SyncThread:2 exits, error code 1
2016-08-24 19:20:19,252 [myid:2] - INFO [SyncThread:2:SyncRequestProcessor@169] - SyncRequestProcessor exited!
2016-08-24 19:20:19,253 [myid:] - INFO [Time-limited test:JUnit4ZKTestRunner$LoggedInvokeMethod@98] - TEST METHOD FAILED testZooKeeperServiceAvailableOnLeader
java.lang.AssertionError: IOException is expected due to error injected to transaction log commit
at org.junit.Assert.fail(Assert.java:88)
at org.apache.zookeeper.test.NonRecoverableErrorTest.testZooKeeperServiceAvailableOnLeader(NonRecoverableErrorTest.java:115)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:79)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.lang.Thread.run(Thread.java:745)
2016-08-24 19:20:19,254 [myid:] - INFO [main:ZKTestCase$1@70] - FAILED testZooKeeperServiceAvailableOnLeader
java.lang.AssertionError: IOException is expected due to error injected to transaction log commit
at org.junit.Assert.fail(Assert.java:88)
at org.apache.zookeeper.test.NonRecoverableErrorTest.testZooKeeperServiceAvailableOnLeader(NonRecoverableErrorTest.java:115)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:79)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.lang.Thread.run(Thread.java:745)
2016-08-24 19:20:19,255 [myid:] - INFO [main:ZKTestCase$1@60] - FINISHED testZooKeeperServiceAvailableOnLeader
{noformat}
flaky, flaky-test 9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
1 year, 21 weeks ago 0|i32rcn:
ZooKeeper ZOOKEEPER-2528

ZooKeeper cluster can become unavailable due to power failures

Bug Open Critical Unresolved Unassigned Ramnatthan Alagappan Ramnatthan Alagappan 24/Aug/16 16:45   09/Mar/17 15:39   3.4.8   server   0 3   A normal ZooKeeper cluster of 3 nodes running on 3 Linux machines. ZooKeeper cluster can become unavailable if power failures happen at certain specific points in time.

Details:

I am running a three-node ZooKeeper cluster. I perform a simple update from a client machine.

When I try to update a value, ZooKeeper creates a new log file (for example, when the current log is fully utilized). First, it creates the file and appends some header information to the newly created log. The system call sequence looks like below:

creat(log.200000001)
append(log.200000001, offset=0, count=16)

Now, if a power failure happens just after the creat of the log file but before the append of the header information, the node simply crashes with an EOF exception. If the same problem occurs at two or more nodes in my three-node cluster, the entire cluster becomes unavailable as the majority of servers have crashed because of the above problem.

A power failure at the same time across multiple nodes may be possible in single data center or single rack deployment scenarios.




9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 30 weeks, 1 day ago 0|i32qxb:
ZooKeeper ZOOKEEPER-2527

Connection watch is not getting cleared when watch is created as part of get and it is fired as part of set and client is closed

Bug Open Major Unresolved Unassigned Rakesh Kumar Singh Rakesh Kumar Singh 24/Aug/16 07:34   28/Jun/17 03:08   3.5.1   server   0 2   Connection watch is not getting cleared when watch is created as part of get and it is fired as part of set and client is closed
Steps to reproduce:-

Configure the Zookeeper in ssl mode and start the same
connect to zookeeper using ./zkCli.sh
Check the watch status as zero.
set watch as below :-
get -w /test
Check the watch it is like below:-
BLR1000007042:/home/Rakesh/Zookeeper/18_Aug/zookeeper-3.5.1-alpha/bin # echo wchs | netcat 10.18.101.80 2181
1 connections watching 1 paths
Total watches:1
let the watch fire as part of below-
set /test hello
Here watch is fired when set is done
Close the client
Check for the watch. It is not zero but 1
BLR1000007042:/home/Rakesh/Zookeeper/18_Aug/zookeeper-3.5.1-alpha/bin # echo wchs | netcat 10.18.101.80 2181
1 connections watching 0 paths
Total watches:0
If we repeat again and again it will keep increasing.
Tried without SSL mode and it is working fine in that mode.
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
2 years, 38 weeks, 1 day ago 0|i32pxz:
ZooKeeper ZOOKEEPER-2526

Add config flag to prohibit connections from clients that don't do Sasl auth

Improvement Open Minor Unresolved Unassigned Irfan Hamid Irfan Hamid 24/Aug/16 00:46   06/Apr/18 21:06   3.4.6 3.4.6 kerberos, security, server   1 5 432000 432000 0% According to ZOOKEEPER-1736 the flag allowSaslFailedClient will allow clients whose Sasl auth has failed the same privileges as a client that does not attempt Sasl, i.e., anonymous login.

It would be nice to have a second property "allowAnonLogin" that defaults to true and allows current behavior. But if it is set to false it disconnects any clients that do not attempt Sasl auth or do not complete it successfully.

The motivation would be to protect a shared ZooKeeper ensemble in a datacenter and reduce the surface area of vulnerability by protecting the service from a resiliency/availability perspective by limiting interaction by anonymous clients. This would also protect against rogue clients that could otherwise deny service by filling up the znode storage in non-ACLed locations.

I'm working off of 3.4.6 source code (that's the one we have deployed internally). This functionality could be implemented by adding a flag ServerCnxn#isAuthenticated that is set to true iff ZooKeeperServer#processSasl() succeeds and which is inspected at every incoming request and the session is closed if auth isn't done and opcode is other than Sasl or Auth:

--- src/java/main/org/apache/zookeeper/server/ServerCnxn.java (revision 1757035)
+++ src/java/main/org/apache/zookeeper/server/ServerCnxn.java (working copy)
@@ -55,6 +55,8 @@
*/
boolean isOldClient = true;

+ boolean isAuthenticated = false;
+
abstract int getSessionTimeout();

abstract void close();



--- src/java/main/org/apache/zookeeper/server/ZooKeeperServer.java (revision 1757035)
+++ src/java/main/org/apache/zookeeper/server/ZooKeeperServer.java (working copy)
@@ -884,11 +892,26 @@
BinaryInputArchive bia = BinaryInputArchive.getArchive(bais);
RequestHeader h = new RequestHeader();
h.deserialize(bia, "header");
// Through the magic of byte buffers, txn will not be
// pointing
// to the start of the txn
incomingBuffer = incomingBuffer.slice();
- if (h.getType() == OpCode.auth) {
+ if (allowAnonLogin == false && cnxn.isAuthenticated == false) {
+ if (!(h.getType() == OpCode.auth ||
+ h.getType() == OpCode.ping ||
+ h.getType() == OpCode.sasl)) {
+ LOG.warn(String.format("Closing client connection %s. OpCode %d received before Sasl authentication was complete and allowAnonLogin=false",
+ cnxn.getRemoteSocketAddress().toString(), h.getType()));
+ ReplyHeader rh = new ReplyHeader(h.getXid(), 0,
+ KeeperException.Code.AUTHFAILED.intValue());
+ cnxn.sendResponse(rh, null, null);
+ cnxn.sendBuffer(ServerCnxnFactory.closeConn);
+ cnxn.disableRecv();
+ }
+ }
@@ -963,6 +986,7 @@
String authorizationID = saslServer.getAuthorizationID();
LOG.info("adding SASL authorization for authorizationID: " + authorizationID);
cnxn.addAuthInfo(new Id("sasl",authorizationID));
+ cnxn.isAuthenticated = true;
}
}
catch (SaslException e) {
0% 0% 432000 432000 newbie, security 9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
Patch
1 year, 49 weeks, 5 days ago 0|i32p33:
ZooKeeper ZOOKEEPER-2525

Throwing Exception at zookeeper server side whenever client is closing the connection

Bug Open Minor Unresolved Unassigned Rakesh Kumar Singh Rakesh Kumar Singh 23/Aug/16 05:36   23/Aug/16 05:36   3.5.1   server   1 2   Throwing Exception at zookeeper server side whenever client is closing the connection

2016-08-17 20:38:09,030 [myid:] - WARN [New I/O worker #4:ClientCnxnSocketNetty$ZKClientHandler@432] - Exception caught: [id: 0xbb7a218d, /0:0:0:0:0:0:0:1:41679 :> localhost/0:0:0:0:0:0:0:1:2181] EXCEPTION: java.nio.channels.ClosedChannelException
java.nio.channels.ClosedChannelException
at org.jboss.netty.handler.ssl.SslHandler$6.run(SslHandler.java:1580)
at org.jboss.netty.channel.socket.ChannelRunnableWrapper.run(ChannelRunnableWrapper.java:40)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.executeInIoThread(AbstractNioWorker.java:71)
at org.jboss.netty.channel.socket.nio.NioWorker.executeInIoThread(NioWorker.java:36)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.executeInIoThread(AbstractNioWorker.java:57)
at org.jboss.netty.channel.socket.nio.NioWorker.executeInIoThread(NioWorker.java:36)
at org.jboss.netty.channel.socket.nio.AbstractNioChannelSink.execute(AbstractNioChannelSink.java:34)
at org.jboss.netty.handler.ssl.SslHandler.channelClosed(SslHandler.java:1566)
at org.jboss.netty.channel.Channels.fireChannelClosed(Channels.java:468)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:376)
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:93)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:109)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:90)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 30 weeks, 2 days ago 0|i32n0n:
ZooKeeper ZOOKEEPER-2524

Start zkServer in ssl mode, start zkCli in non-ssl mode but on ssl port then try to quit at client, it takes almost 30 seconds to quit

Bug Open Major Unresolved Unassigned Rakesh Kumar Singh Rakesh Kumar Singh 23/Aug/16 05:35   23/Aug/16 05:35   3.5.1       0 1   Start zkServer in ssl mode, start zkCli in non-ssl mode but on ssl port then try to quit at client, it takes almost 30 seconds to quit.

Steps to reproduce:-

Configure the details required for SSL in zkServer and zkclient
Make "-Dzookeeper.client.secure=false" for client
Configure the clientPort=2181 and secureClientPort=3181 zoo.cfg file
Start zookeeper server and then client as "zkCli.sh -server :3181
Then quit at client console
It takes almost 30 seconds to quit.

Log at server side is attached.
Log at client side is as below:-

[zk: 10.18.101.80:3181(CONNECTING) 0] quit
2016-08-18 15:02:19,076 [myid:] - INFO [New I/O worker #1:ClientCnxnSocketNetty$ZKClientHandler@377] - channel is disconnected: [id: 0x07b576fd, /10.18.101.80:42228 :> 10.18.101.80/10.18.101.80:3181]
2016-08-18 15:02:19,077 [myid:] - INFO [New I/O worker #1:ClientCnxnSocketNetty@201] - channezkServer.txtl is told closing
2016-08-18 15:02:19,080 [myid:] - INFO [main:ClientCnxnSocketNetty@201] - channel is told closing
2016-08-18 15:02:19,080 [myid:] - INFO [main:ZooKeeper@1110] - Session: 0x0 closed
2016-08-18 15:02:19,080 [myid:] - INFO [main-EventThread:ClientCnxn$EventThread@542] - EventThread shut down for session: 0x0
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 30 weeks, 2 days ago 0|i32n0f:
ZooKeeper ZOOKEEPER-2523

No proper logging when zookeeper failed to start in ssl mode.

Bug Open Minor Unresolved Unassigned Rakesh Kumar Singh Rakesh Kumar Singh 23/Aug/16 05:33   23/Aug/16 05:33   3.5.1   server   0 1   Scenario1 :-

Configure zookeeper for all but configure wrong ssl password
Start the zookeeper server, it starts fine. No logs that zookeeper is not started in ssl mode though it has not started in ssl mode but normal mode.
Try to connect with client, it will failed to connect in ssl port or normal port (as client is started in ssl mode)
Scenario 2:-

Configure the ssl port as 0 and start the server
The log level is in info saying not binding ..
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 30 weeks, 2 days ago 0|i32mzz:
ZooKeeper ZOOKEEPER-2522

Need to document that default "secureClientPort" for Client will be 2181 and not "clientPort" when starting in SSL enable mode

Bug Open Minor Unresolved Unassigned Rakesh Kumar Singh Rakesh Kumar Singh 23/Aug/16 05:12   23/Aug/16 05:12   3.5.1       0 1   By default we configure the clientPort as 2181 in zookeeper and secureClientPort as 3181 in SSL enable zookeeper. But when start the corresponding client (which is in ssl enabled mode) fails to connect because it is trying to connect 2181 in secure mode. 9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 30 weeks, 2 days ago 0|i32myn:
ZooKeeper ZOOKEEPER-2521

space should be truncated while reading password for keystore/truststore which is required to configure while SSL enabled

Bug Open Minor Unresolved Unassigned Rakesh Kumar Singh Rakesh Kumar Singh 23/Aug/16 02:49   05/Feb/20 07:17   3.5.1 3.7.0, 3.5.8 server   0 5   space should be truncated while reading password for keystore/truststore which is required to configure while SSL enabled.
As of now if we configure the password with any heading/trailing space, the zookeeper server will fail to start.
9223372036854775807 No Perforce job exists for this issue. 2 9223372036854775807
3 years, 21 weeks, 1 day ago 0|i32mqn:
ZooKeeper ZOOKEEPER-2520

Zookeeper should accept and handle encrypted password (AES) which is required to be passed for keystore and truststore in case of SSL support

Bug Open Major Unresolved Unassigned Rakesh Kumar Singh Rakesh Kumar Singh 23/Aug/16 02:44   23/Aug/16 02:44   3.5.1   security   0 1   When enable SSL support we need to configure the keystore and truststore details and corresponding password in zkEnv.sh/bat file. Currenlty we can only pass plain password which is not accepted because of security issue.
We should provide the provision to handle the latest encryption mechanism to handle this.
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 30 weeks, 2 days ago 0|i32mq7:
ZooKeeper ZOOKEEPER-2519

zh->state should not be 0 while handle is active

Bug In Progress Major Unresolved Andrew Grasso Andrew Grasso Andrew Grasso 22/Aug/16 08:45   22/Sep/16 14:36   3.4.6   c client   0 3   0 does not correspond to any of the defined states for the zookeeper handle, so a client should not expect to see this value. But in the function {{handle_error}}, we set {{zh->state = 0}}, which a client may then see. Instead, we should set our state to be {{ZOO_CONNECTING_STATE}}.

At some point the code moved away from 0 as a valid state and introduced the defined states. This broke the fix to ZOOKEEPER-800, which checks if state is 0 to know if the handle has been created but has not yet connected. We now use {{ZOO_NOTCONNECTED_STATE}} to mean this, so the check for this in {{zoo_add_auth}} must be changed.

We saw this error in 3.4.6, but I believe it remains present in trunk.
9223372036854775807 No Perforce job exists for this issue. 2 9223372036854775807
3 years, 26 weeks ago 0|i32l9j:
ZooKeeper ZOOKEEPER-2518

Document how to set up tool chains for build and test C client on all platforms

Improvement Open Minor Unresolved Unassigned Michael Han Michael Han 19/Aug/16 20:14   19/Aug/16 20:16       documentation   0 2   Add documentations on how to set up ZooKeeper development environment for C client for all platforms. 9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 30 weeks, 6 days ago 0|i32js7:
ZooKeeper ZOOKEEPER-2517

jute.maxbuffer is ignored

Bug Closed Blocker Fixed Mohammad Arshad Benjamin Jaton Benjamin Jaton 18/Aug/16 19:41   17/May/17 23:43 30/Dec/16 17:02 3.5.2 3.5.3, 3.6.0     1 8   In ClientCnxnSocket.java the parsing of the system property is erroneous:

{code}packetLen = Integer.getInteger(
clientConfig.getProperty(ZKConfig.JUTE_MAXBUFFER),
ZKClientConfig.CLIENT_MAX_PACKET_LENGTH_DEFAULT
);{code}

Javadoc of Integer.getInteger states "The first argument is treated as the name of a system property", whereas here the value of the property is passed.

Instead I believe the author meant to write something like:

{code}packetLen = Integer.parseInt(
clientConfig.getProperty(
ZKConfig.JUTE_MAXBUFFER,
String.valueOf(ZKClientConfig.CLIENT_MAX_PACKET_LENGTH_DEFAULT)
)
);{code}
9223372036854775807 No Perforce job exists for this issue. 5 9223372036854775807
3 years, 11 weeks, 6 days ago 0|i32hzz:
ZooKeeper ZOOKEEPER-2516

C client calculates invalid time interval for pings et al

Bug Open Minor Unresolved Unassigned Hadriel Kaplan Hadriel Kaplan 17/Aug/16 14:43   24/Aug/16 18:27   3.4.0, 3.4.8, 3.5.0, 3.5.1, 3.6.0   c client   0 2   The C-client has a function called {{calculate_interval()}} in {{zookeeper.c}}, whose purpose is to determine the number of milliseconds difference between a start and end time.

Unfortunately its logic is invalid, if the number of microseconds of the end time happens to be less than the number of microseconds of the start time - which it will be about half the time, since the end time could be in the next second interval. Such a case would yield a very big negative number, making the function return an invalid value.

Instead of re-creating the wheel, the {{calculate_interval()}} should use the {{timersub()}} function from {{time.h}} if it's available - if it's not #define'd, then #define it. (it's a macro, and the source code for it is readily available)
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 30 weeks, 1 day ago 0|i32ftj:
ZooKeeper ZOOKEEPER-2515

SessionTrackerImpl non-daemon thread slow to shutdown

Bug Open Major Unresolved Unassigned Paul Millar Paul Millar 17/Aug/16 12:34   17/Aug/16 12:34   3.4.6       1 4   While calling SessionTrackerImpl#shutdown does result in the thread eventually stopping, it takes up to expirationInterval (3 seconds, by default) for the thread to finally die.

Since the thread is not a daemon, this delays the shutdown of any application that makes use of ZooKeeper.

I believe the issue is simple to fix: if the shutdown method notified the thread from within object's monitor then this issue will be resolved.
easyfix 9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 31 weeks, 1 day ago 0|i32fl3:
ZooKeeper ZOOKEEPER-2514

Simplify releasenotes creation for 3.4 branch - consistent with newer branches.

Improvement Closed Critical Fixed Patrick D. Hunt Patrick D. Hunt Patrick D. Hunt 16/Aug/16 17:44   04/Sep/16 01:28 16/Aug/16 19:12 3.4.8 3.4.9 documentation   0 1   ZOOKEEPER-2364 introduced a new process for creating release notes for 3.5 and later branches. Backport this to 3.4 branch in order to make the release manager's life easier (and be consisten with "how to release" page). 9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
3 years, 31 weeks, 2 days ago
Reviewed
0|i32e53:
ZooKeeper ZOOKEEPER-2513

majorChange exceptions during leader sync

Bug Open Critical Unresolved Mohammad Arshad Alexander Shraer Alexander Shraer 13/Aug/16 20:46   05/Sep/16 19:02   3.5.2   server   0 4   In Learner.java there are exceptions being thrown in case majorChange = true, i.e., a reconfig is encountered in the stream of updates from the leader. There may be two problems in the way such exceptions are thrown:
1. important actions, e.g., processTxn, will not be done if an exception is thrown
2. its unclear that the learner will be able to continue where it left off in the process of syncing with the leader, if that sync is interrupted by an exception.

This requires further investigation. Whereas similar code in Follower and Observer is extensively tested, this code in Learner isn't tested as much. We could build on the test case developed in ZOOKEEPER-2172 to make sure this code works properly.
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 28 weeks, 3 days ago 0|i32aov:
ZooKeeper ZOOKEEPER-2512

Allow Jetty dependency to use HTTPS and Basic Auth

Improvement Open Major Unresolved Unassigned Edward Ribeiro Edward Ribeiro 12/Aug/16 14:27   05/Feb/20 07:17   3.5.2 3.7.0, 3.5.8 server   0 1   If ZOOKEEPER-2489 gets committed then it would be nice to allow more flexible configuration of https and basic authentication to JettyAdminServer. JettyAdminServer, security 9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 31 weeks, 6 days ago 0|i329sv:
ZooKeeper ZOOKEEPER-2511

Implement AutoCloseable in ZooKeeper.java

Improvement Closed Major Fixed Abraham Fine Abraham Fine Abraham Fine 11/Aug/16 17:31   17/May/17 23:44 09/Jan/17 16:50   3.5.3, 3.6.0     0 5   As a java developer I would like to be able to use try-with-resource blocks with ZooKeeper objects in order to make closing sessions easier. 9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
3 years, 10 weeks, 3 days ago 0|i3287r:
ZooKeeper ZOOKEEPER-2510

org.apache.zookeeper.server.NettyServerCnxnTest uses wrong import for junit

Bug Open Major Unresolved Ted Dunning Ted Dunning Ted Dunning 10/Aug/16 19:50   05/Feb/20 07:15   3.5.1, 3.5.2 3.7.0, 3.5.8     0 1   junit.framework.Assert is deprecated. The code should use org.junit.Assert instead.

Patch coming shortly.
9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
3 years, 28 weeks ago 0|i326cv:
ZooKeeper ZOOKEEPER-2509

Secure mode leaks memory

Bug Patch Available Major Unresolved Ted Dunning Ted Dunning Ted Dunning 10/Aug/16 00:55   05/Feb/20 07:11   3.5.1, 3.5.2 3.7.0, 3.5.8     0 6   The Netty connection handling logic fails to clean up watches on connection close. This causes memory to leak.

I will have a repro script available soon and a fix. I am not sure how to build a unit test since we would need to build an entire server and generate keys and such. Advice on that appreciated.

9223372036854775807 No Perforce job exists for this issue. 6 9223372036854775807
2 years, 38 weeks, 1 day ago 0|i324iv:
ZooKeeper ZOOKEEPER-2508

Many ZooKeeper tests are flaky because they proceed with zk operation without connecting to ZooKeeper server.

Test Closed Major Fixed Mohammad Arshad Mohammad Arshad Mohammad Arshad 09/Aug/16 09:40   17/May/17 23:44 23/Aug/16 19:37   3.5.3, 3.6.0 tests   0 5   Many ZooKeeper tests are flaky because they proceed with zk operation without connecting to ZooKeeper server.
Recently in our build {{org.apache.zookeeper.server.ZooKeeperServerMainTest.testStandalone()}} failed.
After analyzing we found that it is failed because it is not waiting for ZooKeeper client get connected to server. In this case normally zookeeper client gets connected immediately but if not connected immediately the test case is bound to fail.
Not only ZooKeeperServerMainTest but there are many other classes which have such test cases. This jira is to address all those test cases.
9223372036854775807 No Perforce job exists for this issue. 6 9223372036854775807
3 years, 30 weeks, 1 day ago
Reviewed
0|i3239r:
ZooKeeper ZOOKEEPER-2507

C unit test improvement: line break between 'ZooKeeper server started' and 'Running'

Improvement Closed Minor Fixed Michael Han Michael Han Michael Han 08/Aug/16 16:28   31/Mar/17 05:01 08/Sep/16 10:34 3.4.8, 3.5.2 3.4.10, 3.5.3, 3.6.0 tests   0 2   Currently we have this on C unit test output:
{noformat}
ZooKeeper server startedRunning
......
OK
{noformat}

This is because the zkServer.sh, when eco 'ZooKeeper server started', does not put a line break at the end. It will be clearer for readers of the console output if we fix this by adding a line break in between, so we can separate the script output and the test output. After the fix the output would look like:

{noformat}
ZooKeeper server started
Running
.....
OK
{noformat}
9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
3 years, 28 weeks ago
Reviewed
0|i321yv:
ZooKeeper ZOOKEEPER-2506

Document use of read-only mode

Improvement Open Minor Unresolved Benjamin Jaton Flavio Paiva Junqueira Flavio Paiva Junqueira 06/Aug/16 16:11   05/Feb/20 07:16     3.7.0, 3.5.8 documentation   0 2   It would be a good addition to the documentation a simple example of how to code an application to use read-only mode. 9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
3 years, 32 weeks, 5 days ago 0|i3203j:
ZooKeeper ZOOKEEPER-2505

Use shared library instead of static library in C client unit test

Improvement Closed Minor Fixed Michael Han Michael Han Michael Han 05/Aug/16 20:05   17/May/17 23:43 23/Aug/16 19:05 3.5.2 3.5.3, 3.6.0 c client   0 3   Currently we are statically linking c unit tests to ZK client library - we should use shared library instead as there seems no particular reason to stick to static library, plus one benefit of using shared library is that would allow us to use overrides function calls from standard libraries at link time so we can simulate the wrap option for ld linker on os x. 9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
3 years, 30 weeks, 1 day ago
Reviewed
0|i31zrj:
ZooKeeper ZOOKEEPER-2504

Enforce that server ids are unique in a cluster

Bug Patch Available Major Unresolved Michael Han Dan Benediktson Dan Benediktson 05/Aug/16 18:48   23/Feb/19 05:11           0 5   The leader will happily accept connections from learners that have the same server id (i.e., due to misconfiguration). This can lead to various issues including non-unique session_ids being generated by these servers.

The leader can enforce that all learners come in with unique server IDs; if a learner attempts to connect with an id that is already in use, it should be denied.
9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
1 year, 3 weeks, 5 days ago Enforces ZAB protocol version 0x10000 on the leader from all connecting learners, thus must be upgraded from 3.4.0 or higher. 0|i31znj:
ZooKeeper ZOOKEEPER-2503

do a hard constraints on the number of myid which must between 0 and 255

Bug Open Major Unresolved maoling Michael Han Michael Han 05/Aug/16 12:01   21/Apr/19 07:06   3.4.9, 3.5.2, 3.4.11   server   0 8 0 9000   In ZK documentation, we have:

"The myid file consists of a single line containing only the text of that machine's id. So myid of server 1 would contain the text "1" and nothing else. The id must be unique within the ensemble and should have a value between 1 and 255."

This however is not enforced in code, which should be fixed either in documentation that we remove the restriction of the range 1-255 or in code we enforce such constraint.
Discussion thread:
[http://zookeeper-user.578899.n2.nabble.com/Is-myid-actually-limited-to-1-255-td7581270.html]

 
100% 100% 9000 0 pull-request-available 9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
1 year, 3 weeks, 4 days ago 0|i31yy7:
ZooKeeper ZOOKEEPER-2502

Flaky Test: org.apache.zookeeper.server.quorum.CnxManagerTest.testCnxFromFutureVersion

Test Closed Major Fixed Michael Han Michael Han Michael Han 05/Aug/16 11:45   31/Mar/17 05:01 06/Sep/16 13:24 3.4.9 3.4.10 tests   0 1   {noformat}
Error Message

Broken pipe
Stacktrace

java.io.IOException: Broken pipe
at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
at sun.nio.ch.IOUtil.write(IOUtil.java:65)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:466)
at java.nio.channels.Channels.writeFullyImpl(Channels.java:78)
at java.nio.channels.Channels.writeFully(Channels.java:98)
at java.nio.channels.Channels.access$000(Channels.java:61)
at java.nio.channels.Channels$1.write(Channels.java:174)
at java.io.OutputStream.write(OutputStream.java:75)
at java.nio.channels.Channels$1.write(Channels.java:155)
at java.io.DataOutputStream.writeInt(DataOutputStream.java:198)
at org.apache.zookeeper.server.quorum.CnxManagerTest.testCnxFromFutureVersion(CnxManagerTest.java:318)
at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:55)
Standard Output

2016-07-12 22:34:46,623 [myid:] - INFO [main:ZKTestCase$1@50] - STARTING testCnxFromFutureVersion
2016-07-12 22:34:46,627 [myid:] - INFO [main:PortAssignment@32] - assigning port 11221
2016-07-12 22:34:46,630 [myid:] - INFO [main:PortAssignment@32] - assigning port 11222
2016-07-12 22:34:46,631 [myid:] - INFO [main:PortAssignment@32] - assigning port 11223
2016-07-12 22:34:46,643 [myid:] - INFO [main:QuorumPeer$QuorumServer@149] - Resolved hostname: 0.0.0.0 to address: /0.0.0.0
2016-07-12 22:34:46,658 [myid:] - INFO [main:PortAssignment@32] - assigning port 11224
2016-07-12 22:34:46,658 [myid:] - INFO [main:PortAssignment@32] - assigning port 11225
2016-07-12 22:34:46,659 [myid:] - INFO [main:PortAssignment@32] - assigning port 11226
2016-07-12 22:34:46,659 [myid:] - INFO [main:QuorumPeer$QuorumServer@149] - Resolved hostname: 0.0.0.0 to address: /0.0.0.0
2016-07-12 22:34:46,659 [myid:] - INFO [main:PortAssignment@32] - assigning port 11227
2016-07-12 22:34:46,659 [myid:] - INFO [main:PortAssignment@32] - assigning port 11228
2016-07-12 22:34:46,659 [myid:] - INFO [main:PortAssignment@32] - assigning port 11229
2016-07-12 22:34:46,660 [myid:] - INFO [main:QuorumPeer$QuorumServer@149] - Resolved hostname: 0.0.0.0 to address: /0.0.0.0
2016-07-12 22:34:46,660 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@53] - RUNNING TEST METHOD testCnxFromFutureVersion
2016-07-12 22:34:46,672 [myid:] - INFO [main:NIOServerCnxnFactory@89] - binding to port 0.0.0.0/0.0.0.0:11225
2016-07-12 22:34:46,692 [myid:] - INFO [main:CnxManagerTest@301] - Election port: 11226
2016-07-12 22:34:46,692 [myid:] - INFO [ListenerThread:QuorumCnxManager$Listener@534] - My election bind port: /0.0.0.0:11226
2016-07-12 22:34:47,696 [myid:] - INFO [/0.0.0.0:11226:QuorumCnxManager$Listener@541] - Received connection request /140.211.11.27:48724
2016-07-12 22:34:49,700 [myid:] - WARN [/0.0.0.0:11226:QuorumCnxManager@274] - Exception reading or writing challenge: java.net.SocketTimeoutException: Read timed out
2016-07-12 22:34:52,700 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@74] - TEST METHOD FAILED testCnxFromFutureVersion
java.io.IOException: Broken pipe
at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
at sun.nio.ch.IOUtil.write(IOUtil.java:65)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:466)
at java.nio.channels.Channels.writeFullyImpl(Channels.java:78)
at java.nio.channels.Channels.writeFully(Channels.java:98)
at java.nio.channels.Channels.access$000(Channels.java:61)
at java.nio.channels.Channels$1.write(Channels.java:174)
at java.io.OutputStream.write(OutputStream.java:75)
at java.nio.channels.Channels$1.write(Channels.java:155)
at java.io.DataOutputStream.writeInt(DataOutputStream.java:198)
at org.apache.zookeeper.server.quorum.CnxManagerTest.testCnxFromFutureVersion(CnxManagerTest.java:318)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:55)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
at org.junit.rules.TestWatchman$1.evaluate(TestWatchman.java:48)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:76)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184)
at org.junit.runners.ParentRunner.run(ParentRunner.java:236)
at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:39)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:518)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1052)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:906)
2016-07-12 22:34:52,705 [myid:] - INFO [main:ZKTestCase$1@65] - FAILED testCnxFromFutureVersion
java.io.IOException: Broken pipe
at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
at sun.nio.ch.IOUtil.write(IOUtil.java:65)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:466)
at java.nio.channels.Channels.writeFullyImpl(Channels.java:78)
at java.nio.channels.Channels.writeFully(Channels.java:98)
at java.nio.channels.Channels.access$000(Channels.java:61)
at java.nio.channels.Channels$1.write(Channels.java:174)
at java.io.OutputStream.write(OutputStream.java:75)
at java.nio.channels.Channels$1.write(Channels.java:155)
at java.io.DataOutputStream.writeInt(DataOutputStream.java:198)
at org.apache.zookeeper.server.quorum.CnxManagerTest.testCnxFromFutureVersion(CnxManagerTest.java:318)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:55)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
at org.junit.rules.TestWatchman$1.evaluate(TestWatchman.java:48)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:76)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184)
at org.junit.runners.ParentRunner.run(ParentRunner.java:236)
at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:39)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:518)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1052)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:906)
2016-07-12 22:34:52,706 [myid:] - INFO [main:ZKTestCase$1@55] - FINISHED testCnxFromFutureVersion
2016-07-12 22:34:52,720 [myid:] - INFO [main:ZKTestCase$1@50] - STARTING testSocketTimeout
2016-07-12 22:34:52,720 [myid:] - INFO [main:PortAssignment@32] - assigning port 11230
2016-07-12 22:34:52,720 [myid:] - INFO [main:PortAssignment@32] - assigning port 11231
2016-07-12 22:34:52,720 [myid:] - INFO [main:PortAssignment@32] - assigning port 11232
2016-07-12 22:34:52,721 [myid:] - INFO [main:QuorumPeer$QuorumServer@149] - Resolved hostname: 0.0.0.0 to address: /0.0.0.0
2016-07-12 22:34:52,721 [myid:] - INFO [main:PortAssignment@32] - assigning port 11233
2016-07-12 22:34:52,722 [myid:] - INFO [main:PortAssignment@32] - assigning port 11234
2016-07-12 22:34:52,722 [myid:] - INFO [main:PortAssignment@32] - assigning port 11235
2016-07-12 22:34:52,722 [myid:] - INFO [main:QuorumPeer$QuorumServer@149] - Resolved hostname: 0.0.0.0 to address: /0.0.0.0
2016-07-12 22:34:52,723 [myid:] - INFO [main:PortAssignment@32] - assigning port 11236
2016-07-12 22:34:52,723 [myid:] - INFO [main:PortAssignment@32] - assigning port 11237
2016-07-12 22:34:52,723 [myid:] - INFO [main:PortAssignment@32] - assigning port 11238
2016-07-12 22:34:52,724 [myid:] - INFO [main:QuorumPeer$QuorumServer@149] - Resolved hostname: 0.0.0.0 to address: /0.0.0.0
2016-07-12 22:34:52,724 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@53] - RUNNING TEST METHOD testSocketTimeout
2016-07-12 22:34:52,725 [myid:] - INFO [main:NIOServerCnxnFactory@89] - binding to port 0.0.0.0/0.0.0.0:11234
2016-07-12 22:34:52,726 [myid:] - INFO [main:CnxManagerTest@370] - Election port: 11235
2016-07-12 22:34:52,726 [myid:] - INFO [ListenerThread:QuorumCnxManager$Listener@534] - My election bind port: /0.0.0.0:11235
2016-07-12 22:34:53,729 [myid:] - INFO [/0.0.0.0:11235:QuorumCnxManager$Listener@541] - Received connection request /140.211.11.27:34496
2016-07-12 22:34:57,734 [myid:] - WARN [/0.0.0.0:11235:QuorumCnxManager@274] - Exception reading or writing challenge: java.net.SocketTimeoutException: Read timed out
2016-07-12 22:34:57,734 [myid:] - WARN [main:QuorumCnxManager@274] - Exception reading or writing challenge: java.io.EOFException
2016-07-12 22:34:57,734 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@58] - Memory used 3153
2016-07-12 22:34:57,735 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@63] - Number of threads 6
2016-07-12 22:34:57,735 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@78] - FINISHED TEST METHOD testSocketTimeout
2016-07-12 22:34:57,735 [myid:] - INFO [main:ZKTestCase$1@60] - SUCCEEDED testSocketTimeout
2016-07-12 22:34:57,735 [myid:] - INFO [main:ZKTestCase$1@55] - FINISHED testSocketTimeout
2016-07-12 22:34:57,735 [myid:] - INFO [main:ZKTestCase$1@50] - STARTING testWorkerThreads
2016-07-12 22:34:57,736 [myid:] - INFO [main:PortAssignment@32] - assigning port 11239
2016-07-12 22:34:57,736 [myid:] - INFO [main:PortAssignment@32] - assigning port 11240
2016-07-12 22:34:57,736 [myid:] - INFO [main:PortAssignment@32] - assigning port 11241
2016-07-12 22:34:57,736 [myid:] - INFO [main:QuorumPeer$QuorumServer@149] - Resolved hostname: 0.0.0.0 to address: /0.0.0.0
2016-07-12 22:34:57,736 [myid:] - INFO [main:PortAssignment@32] - assigning port 11242
2016-07-12 22:34:57,737 [myid:] - INFO [main:PortAssignment@32] - assigning port 11243
2016-07-12 22:34:57,737 [myid:] - INFO [main:PortAssignment@32] - assigning port 11244
2016-07-12 22:34:57,737 [myid:] - INFO [main:QuorumPeer$QuorumServer@149] - Resolved hostname: 0.0.0.0 to address: /0.0.0.0
2016-07-12 22:34:57,737 [myid:] - INFO [main:PortAssignment@32] - assigning port 11245
2016-07-12 22:34:57,737 [myid:] - INFO [main:PortAssignment@32] - assigning port 11246
2016-07-12 22:34:57,737 [myid:] - INFO [main:PortAssignment@32] - assigning port 11247
2016-07-12 22:34:57,737 [myid:] - INFO [main:QuorumPeer$QuorumServer@149] - Resolved hostname: 0.0.0.0 to address: /0.0.0.0
2016-07-12 22:34:57,738 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@53] - RUNNING TEST METHOD testWorkerThreads
2016-07-12 22:34:57,738 [myid:] - INFO [main:NIOServerCnxnFactory@89] - binding to port 0.0.0.0/0.0.0.0:11240
2016-07-12 22:34:57,784 [myid:] - INFO [main:CnxManagerTest@393] - Starting peer 0
2016-07-12 22:34:57,789 [myid:] - INFO [main:QuorumPeer@533] - currentEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
2016-07-12 22:34:57,905 [myid:] - INFO [main:QuorumPeer@548] - acceptedEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
2016-07-12 22:34:57,945 [myid:] - INFO [ListenerThread:QuorumCnxManager$Listener@534] - My election bind port: /0.0.0.0:11241
2016-07-12 22:34:57,953 [myid:] - INFO [main:NIOServerCnxnFactory@89] - binding to port 0.0.0.0/0.0.0.0:11243
2016-07-12 22:34:57,954 [myid:] - INFO [main:CnxManagerTest@393] - Starting peer 1
2016-07-12 22:34:57,954 [myid:] - INFO [main:QuorumPeer@533] - currentEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
2016-07-12 22:34:58,024 [myid:] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11240:QuorumPeer@774] - LOOKING
2016-07-12 22:34:58,025 [myid:] - INFO [main:QuorumPeer@548] - acceptedEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
2016-07-12 22:34:58,025 [myid:] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11240:FastLeaderElection@818] - New election. My id = 0, proposed zxid=0x0
2016-07-12 22:34:58,028 [myid:] - INFO [WorkerReceiver[myid=0]:FastLeaderElection@600] - Notification: 1 (message format version), 0 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 0 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-12 22:34:58,028 [myid:] - WARN [WorkerSender[myid=0]:QuorumCnxManager@400] - Cannot open channel to 1 at election address /0.0.0.0:11244
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:345)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:204)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:381)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:354)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:452)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:433)
at java.lang.Thread.run(Thread.java:744)
2016-07-12 22:34:58,029 [myid:] - INFO [WorkerSender[myid=0]:QuorumPeer$QuorumServer@149] - Resolved hostname: 0.0.0.0 to address: /0.0.0.0
2016-07-12 22:34:58,030 [myid:] - WARN [WorkerSender[myid=0]:QuorumCnxManager@400] - Cannot open channel to 2 at election address /0.0.0.0:11247
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:345)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:204)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:381)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:354)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:452)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:433)
at java.lang.Thread.run(Thread.java:744)
2016-07-12 22:34:58,030 [myid:] - INFO [WorkerSender[myid=0]:QuorumPeer$QuorumServer@149] - Resolved hostname: 0.0.0.0 to address: /0.0.0.0
2016-07-12 22:34:58,050 [myid:] - INFO [ListenerThread:QuorumCnxManager$Listener@534] - My election bind port: /0.0.0.0:11244
2016-07-12 22:34:58,052 [myid:] - INFO [main:NIOServerCnxnFactory@89] - binding to port 0.0.0.0/0.0.0.0:11246
2016-07-12 22:34:58,052 [myid:] - INFO [main:CnxManagerTest@393] - Starting peer 2
2016-07-12 22:34:58,053 [myid:] - INFO [main:QuorumPeer@533] - currentEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
2016-07-12 22:34:58,057 [myid:] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11243:QuorumPeer@774] - LOOKING
2016-07-12 22:34:58,058 [myid:] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11243:FastLeaderElection@818] - New election. My id = 1, proposed zxid=0x0
2016-07-12 22:34:58,058 [myid:] - INFO [/0.0.0.0:11241:QuorumCnxManager$Listener@541] - Received connection request /140.211.11.27:46871
2016-07-12 22:34:58,063 [myid:] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message format version), 1 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-12 22:34:58,064 [myid:] - INFO [WorkerReceiver[myid=0]:FastLeaderElection@600] - Notification: 1 (message format version), 1 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-12 22:34:58,064 [myid:] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message format version), 0 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 0 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-12 22:34:58,065 [myid:] - INFO [WorkerReceiver[myid=0]:FastLeaderElection@600] - Notification: 1 (message format version), 1 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 0 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-12 22:34:58,065 [myid:] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message format version), 1 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 0 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-12 22:34:58,065 [myid:] - WARN [WorkerSender[myid=1]:QuorumCnxManager@400] - Cannot open channel to 2 at election address /0.0.0.0:11247
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:345)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:204)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:381)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:354)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:452)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:433)
at java.lang.Thread.run(Thread.java:744)
2016-07-12 22:34:58,065 [myid:] - WARN [WorkerSender[myid=0]:QuorumCnxManager@400] - Cannot open channel to 2 at election address /0.0.0.0:11247
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:345)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:204)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:381)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:354)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:452)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:433)
at java.lang.Thread.run(Thread.java:744)
2016-07-12 22:34:58,066 [myid:] - INFO [WorkerSender[myid=1]:QuorumPeer$QuorumServer@149] - Resolved hostname: 0.0.0.0 to address: /0.0.0.0
2016-07-12 22:34:58,066 [myid:] - INFO [WorkerSender[myid=0]:QuorumPeer$QuorumServer@149] - Resolved hostname: 0.0.0.0 to address: /0.0.0.0
2016-07-12 22:34:58,084 [myid:] - INFO [main:QuorumPeer@548] - acceptedEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
2016-07-12 22:34:58,133 [myid:] - INFO [ListenerThread:QuorumCnxManager$Listener@534] - My election bind port: /0.0.0.0:11247
2016-07-12 22:34:58,136 [myid:] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11246:QuorumPeer@774] - LOOKING
2016-07-12 22:34:58,136 [myid:] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11246:FastLeaderElection@818] - New election. My id = 2, proposed zxid=0x0
2016-07-12 22:34:58,137 [myid:] - INFO [/0.0.0.0:11241:QuorumCnxManager$Listener@541] - Received connection request /140.211.11.27:46874
2016-07-12 22:34:58,138 [myid:] - INFO [/0.0.0.0:11244:QuorumCnxManager$Listener@541] - Received connection request /140.211.11.27:33135
2016-07-12 22:34:58,141 [myid:] - INFO [WorkerReceiver[myid=0]:FastLeaderElection@600] - Notification: 1 (message format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-12 22:34:58,143 [myid:] - INFO [WorkerReceiver[myid=0]:FastLeaderElection@600] - Notification: 1 (message format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 0 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-12 22:34:58,143 [myid:] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-12 22:34:58,143 [myid:] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@600] - Notification: 1 (message format version), 1 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 0 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-12 22:34:58,144 [myid:] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 0 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-12 22:34:58,144 [myid:] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@600] - Notification: 1 (message format version), 1 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-12 22:34:58,144 [myid:] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-12 22:34:58,144 [myid:] - INFO [WorkerReceiver[myid=0]:FastLeaderElection@600] - Notification: 1 (message format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-12 22:34:58,144 [myid:] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@600] - Notification: 1 (message format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-12 22:34:58,145 [myid:] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@600] - Notification: 1 (message format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 0 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-12 22:34:58,145 [myid:] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@600] - Notification: 1 (message format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-12 22:34:58,345 [myid:] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11243:QuorumPeer@844] - FOLLOWING
2016-07-12 22:34:58,345 [myid:] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11240:QuorumPeer@844] - FOLLOWING
2016-07-12 22:34:58,346 [myid:] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11246:QuorumPeer@856] - LEADING
2016-07-12 22:34:58,349 [myid:] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11243:Learner@86] - TCP NoDelay set to: true
2016-07-12 22:34:58,350 [myid:] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11246:Leader@59] - TCP NoDelay set to: true
2016-07-12 22:34:58,354 [myid:] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11243:Environment@100] - Server environment:zookeeper.version=3.4.9-SNAPSHOT-1752356, built on 07/12/2016 22:30 GMT
2016-07-12 22:34:58,354 [myid:] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11243:Environment@100] - Server environment:host.name=hemera.apache.org
2016-07-12 22:34:58,354 [myid:] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11243:Environment@100] - Server environment:java.version=1.8.0
2016-07-12 22:34:58,354 [myid:] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11243:Environment@100] - Server environment:java.vendor=Oracle Corporation
2016-07-12 22:34:58,354 [myid:] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11243:Environment@100] - Server environment:java.home=/x1/jenkins/jenkins-slave/tools/hudson.model.JDK/jdk-1.8.0/jre
2016-07-12 22:34:58,354 [myid:] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11243:Environment@100] - Server environment:java.class.path=/x1/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk8/branch-3.4/build/test/classes:/x1/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk8/branch-3.4/build/test/lib/antlr-2.7.6.jar:/x1/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk8/branch-3.4/build/test/lib/checkstyle-5.0.jar:/x1/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk8/branch-3.4/build/test/lib/commons-beanutils-core-1.7.0.jar:/x1/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk8/branch-3.4/build/test/lib/commons-cli-1.0.jar:/x1/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk8/branch-3.4/build/test/lib/commons-collections-3.2.2.jar:/x1/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk8/branch-3.4/build/test/lib/commons-lang-1.0.jar:/x1/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk8/branch-3.4/build/test/lib/commons-logging-1.0.3.jar:/x1/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk8/branch-3.4/build/test/lib/google-collections-0.9.jar:/x1/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk8/branch-3.4/build/test/lib/junit-4.8.1.jar:/x1/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk8/branch-3.4/build/test/lib/mockito-all-1.8.2.jar:/x1/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk8/branch-3.4/build/classes:/x1/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk8/branch-3.4/src/java/lib/ivy-2.4.0.jar:/home/jenkins/tools/ant/latest/lib/ant.jar:/x1/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk8/branch-3.4/build/lib/jline-0.9.94.jar:/x1/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk8/branch-3.4/build/lib/log4j-1.2.16.jar:/x1/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk8/branch-3.4/build/lib/netty-3.10.5.Final.jar:/x1/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk8/branch-3.4/build/lib/slf4j-api-1.6.1.jar:/x1/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk8/branch-3.4/build/lib/slf4j-log4j12-1.6.1.jar:/home/jenkins/tools/clover/latest/lib/clover.jar:/x1/jenkins/tools/ant/apache-ant-1.8.2/lib/ant-launcher.jar:/home/jenkins/tools/ant/latest/lib/ant-junit.jar:/home/jenkins/tools/ant/latest/lib/ant-junit4.jar
2016-07-12 22:34:58,355 [myid:] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11243:Environment@100] - Server environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2016-07-12 22:34:58,355 [myid:] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11243:Environment@100] - Server environment:java.io.tmpdir=/tmp
2016-07-12 22:34:58,355 [myid:] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11243:Environment@100] - Server environment:java.compiler=<NA>
2016-07-12 22:34:58,355 [myid:] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11243:Environment@100] - Server environment:os.name=Linux
2016-07-12 22:34:58,355 [myid:] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11243:Environment@100] - Server environment:os.arch=amd64
2016-07-12 22:34:58,355 [myid:] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11243:Environment@100] - Server environment:os.version=3.2.0-104-generic
2016-07-12 22:34:58,355 [myid:] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11243:Environment@100] - Server environment:user.name=jenkins
2016-07-12 22:34:58,355 [myid:] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11243:Environment@100] - Server environment:user.home=/home/jenkins
2016-07-12 22:34:58,355 [myid:] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11243:Environment@100] - Server environment:user.dir=/x1/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk8/branch-3.4
2016-07-12 22:34:58,358 [myid:] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11243:ZooKeeperServer@170] - Created server with tickTime 1000 minSessionTimeout 2000 maxSessionTimeout 20000 datadir /x1/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk8/branch-3.4/build/test/tmp/test6977370939981061612.junit.dir/version-2 snapdir /x1/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk8/branch-3.4/build/test/tmp/test6977370939981061612.junit.dir/version-2
2016-07-12 22:34:58,358 [myid:] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11246:ZooKeeperServer@170] - Created server with tickTime 1000 minSessionTimeout 2000 maxSessionTimeout 20000 datadir /x1/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk8/branch-3.4/build/test/tmp/test4996523116536588259.junit.dir/version-2 snapdir /x1/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk8/branch-3.4/build/test/tmp/test4996523116536588259.junit.dir/version-2
2016-07-12 22:34:58,358 [myid:] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11240:ZooKeeperServer@170] - Created server with tickTime 1000 minSessionTimeout 2000 maxSessionTimeout 20000 datadir /x1/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk8/branch-3.4/build/test/tmp/test8108741215991761367.junit.dir/version-2 snapdir /x1/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk8/branch-3.4/build/test/tmp/test8108741215991761367.junit.dir/version-2
2016-07-12 22:34:58,359 [myid:] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11243:Follower@63] - FOLLOWING - LEADER ELECTION TOOK - 301
2016-07-12 22:34:58,359 [myid:] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11240:Follower@63] - FOLLOWING - LEADER ELECTION TOOK - 334
2016-07-12 22:34:58,365 [myid:] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11246:Leader@361] - LEADING - LEADER ELECTION TOOK - 224
2016-07-12 22:34:58,370 [myid:] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11240:QuorumPeer$QuorumServer@149] - Resolved hostname: 0.0.0.0 to address: /0.0.0.0
2016-07-12 22:34:58,373 [myid:] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11243:QuorumPeer$QuorumServer@149] - Resolved hostname: 0.0.0.0 to address: /0.0.0.0
2016-07-12 22:34:58,383 [myid:] - INFO [LearnerHandler-/140.211.11.27:45223:LearnerHandler@329] - Follower sid: 1 : info : org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@4e46c644
2016-07-12 22:34:58,383 [myid:] - INFO [LearnerHandler-/140.211.11.27:45222:LearnerHandler@329] - Follower sid: 0 : info : org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@23da3d
2016-07-12 22:34:58,528 [myid:] - INFO [LearnerHandler-/140.211.11.27:45222:LearnerHandler@384] - Synchronizing with Follower sid: 0 maxCommittedLog=0x0 minCommittedLog=0x0 peerLastZxid=0x0
2016-07-12 22:34:58,528 [myid:] - INFO [LearnerHandler-/140.211.11.27:45223:LearnerHandler@384] - Synchronizing with Follower sid: 1 maxCommittedLog=0x0 minCommittedLog=0x0 peerLastZxid=0x0
2016-07-12 22:34:58,529 [myid:] - INFO [LearnerHandler-/140.211.11.27:45222:LearnerHandler@393] - leader and follower are in sync, zxid=0x0
2016-07-12 22:34:58,529 [myid:] - INFO [LearnerHandler-/140.211.11.27:45223:LearnerHandler@393] - leader and follower are in sync, zxid=0x0
2016-07-12 22:34:58,529 [myid:] - INFO [LearnerHandler-/140.211.11.27:45223:LearnerHandler@458] - Sending DIFF
2016-07-12 22:34:58,529 [myid:] - INFO [LearnerHandler-/140.211.11.27:45222:LearnerHandler@458] - Sending DIFF
2016-07-12 22:34:58,530 [myid:] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11243:Learner@326] - Getting a diff from the leader 0x0
2016-07-12 22:34:58,530 [myid:] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11240:Learner@326] - Getting a diff from the leader 0x0
2016-07-12 22:34:58,534 [myid:] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11243:FileTxnSnapLog@240] - Snapshotting: 0x0 to /x1/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk8/branch-3.4/build/test/tmp/test6977370939981061612.junit.dir/version-2/snapshot.0
2016-07-12 22:34:58,534 [myid:] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11240:FileTxnSnapLog@240] - Snapshotting: 0x0 to /x1/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk8/branch-3.4/build/test/tmp/test8108741215991761367.junit.dir/version-2/snapshot.0
2016-07-12 22:34:58,623 [myid:] - INFO [LearnerHandler-/140.211.11.27:45222:LearnerHandler@518] - Received NEWLEADER-ACK message from 0
2016-07-12 22:34:58,623 [myid:] - INFO [LearnerHandler-/140.211.11.27:45223:LearnerHandler@518] - Received NEWLEADER-ACK message from 1
2016-07-12 22:34:58,623 [myid:] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11246:Leader@946] - Have quorum of supporters, sids: [ 0,2 ]; starting up and setting last processed zxid: 0x100000000
2016-07-12 22:34:58,634 [myid:] - INFO [main:CnxManagerTest@405] - Round 0, halting peer 0
2016-07-12 22:34:58,634 [myid:] - INFO [main:Follower@166] - shutdown called
java.lang.Exception: shutdown Follower
at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:166)
at org.apache.zookeeper.server.quorum.QuorumPeer.shutdown(QuorumPeer.java:891)
at org.apache.zookeeper.server.quorum.CnxManagerTest.testWorkerThreads(CnxManagerTest.java:407)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:55)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
at org.junit.rules.TestWatchman$1.evaluate(TestWatchman.java:48)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:76)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184)
at org.junit.runners.ParentRunner.run(ParentRunner.java:236)
at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:39)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:518)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1052)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:906)
2016-07-12 22:34:58,635 [myid:] - INFO [main:FollowerZooKeeperServer@140] - Shutting down
2016-07-12 22:34:58,645 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11240:NIOServerCnxnFactory@219] - NIOServerCnxn factory exited run method
2016-07-12 22:34:58,645 [myid:] - INFO [main:FollowerZooKeeperServer@140] - Shutting down
2016-07-12 22:34:58,647 [myid:] - INFO [main:ZooKeeperServer@469] - shutting down
2016-07-12 22:34:58,647 [myid:] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11240:Follower@166] - shutdown called
java.lang.Exception: shutdown Follower
at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:166)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:850)
2016-07-12 22:34:58,648 [myid:] - INFO [main:FollowerRequestProcessor@107] - Shutting down
2016-07-12 22:34:58,648 [myid:] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11240:FollowerZooKeeperServer@140] - Shutting down
2016-07-12 22:34:58,648 [myid:] - INFO [FollowerRequestProcessor:0:FollowerRequestProcessor@97] - FollowerRequestProcessor exited loop!
2016-07-12 22:34:58,648 [myid:] - INFO [main:CommitProcessor@184] - Shutting down
2016-07-12 22:34:58,649 [myid:] - INFO [main:FinalRequestProcessor@402] - shutdown of request processor complete
2016-07-12 22:34:58,649 [myid:] - INFO [CommitProcessor:0:CommitProcessor@153] - CommitProcessor exited loop!
2016-07-12 22:34:58,650 [myid:] - INFO [main:SyncRequestProcessor@209] - Shutting down
2016-07-12 22:34:58,650 [myid:] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11240:SyncRequestProcessor@209] - Shutting down
2016-07-12 22:34:58,651 [myid:] - INFO [SyncThread:0:SyncRequestProcessor@187] - SyncRequestProcessor exited!
2016-07-12 22:34:58,651 [myid:] - WARN [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11240:QuorumPeer@874] - QuorumPeer main thread exited
2016-07-12 22:34:58,651 [myid:] - ERROR [/0.0.0.0:11241:QuorumCnxManager$Listener@547] - Exception while listening
java.net.SocketException: Socket closed
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:404)
at java.net.ServerSocket.implAccept(ServerSocket.java:545)
at java.net.ServerSocket.accept(ServerSocket.java:513)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$Listener.run(QuorumCnxManager.java:539)
2016-07-12 22:34:58,652 [myid:] - WARN [SendWorker:1:QuorumCnxManager$SendWorker@727] - Interrupted while waiting for message on queue
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:879)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:65)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:715)
2016-07-12 22:34:58,653 [myid:] - WARN [RecvWorker:2:QuorumCnxManager$RecvWorker@810] - Connection broken for id 2, my id = 0, error =
java.net.SocketException: Socket closed
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:150)
at java.net.SocketInputStream.read(SocketInputStream.java:121)
at java.net.SocketInputStream.read(SocketInputStream.java:203)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:795)
2016-07-12 22:34:58,654 [myid:] - WARN [RecvWorker:2:QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker
2016-07-12 22:34:58,653 [myid:] - WARN [RecvWorker:0:QuorumCnxManager$RecvWorker@810] - Connection broken for id 0, my id = 2, error =
java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:392)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:795)
2016-07-12 22:34:58,653 [myid:] - WARN [SendWorker:2:QuorumCnxManager$SendWorker@727] - Interrupted while waiting for message on queue
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:879)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:65)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:715)
2016-07-12 22:34:58,656 [myid:] - WARN [SendWorker:2:QuorumCnxManager$SendWorker@736] - Send worker leaving thread
2016-07-12 22:34:58,652 [myid:] - WARN [RecvWorker:0:QuorumCnxManager$RecvWorker@810] - Connection broken for id 0, my id = 1, error =
java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:392)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:795)
2016-07-12 22:34:58,656 [myid:] - WARN [RecvWorker:0:QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker
2016-07-12 22:34:58,652 [myid:] - WARN [RecvWorker:1:QuorumCnxManager$RecvWorker@810] - Connection broken for id 1, my id = 0, error =
java.net.SocketException: Socket closed
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:150)
at java.net.SocketInputStream.read(SocketInputStream.java:121)
at java.net.SocketInputStream.read(SocketInputStream.java:203)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:795)
2016-07-12 22:34:58,657 [myid:] - WARN [RecvWorker:1:QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker
2016-07-12 22:34:58,657 [myid:] - WARN [SendWorker:0:QuorumCnxManager$SendWorker@727] - Interrupted while waiting for message on queue
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:879)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:65)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:715)
2016-07-12 22:34:58,658 [myid:] - WARN [SendWorker:0:QuorumCnxManager$SendWorker@736] - Send worker leaving thread
2016-07-12 22:34:58,655 [myid:] - WARN [RecvWorker:0:QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker
2016-07-12 22:34:58,654 [myid:] - WARN [SendWorker:1:QuorumCnxManager$SendWorker@736] - Send worker leaving thread
2016-07-12 22:34:58,659 [myid:] - WARN [SendWorker:0:QuorumCnxManager$SendWorker@727] - Interrupted while waiting for message on queue
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:879)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:65)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:715)
2016-07-12 22:34:58,659 [myid:] - WARN [SendWorker:0:QuorumCnxManager$SendWorker@736] - Send worker leaving thread
2016-07-12 22:34:59,154 [myid:] - INFO [main:NIOServerCnxnFactory@89] - binding to port 0.0.0.0/0.0.0.0:11240
2016-07-12 22:34:59,155 [myid:] - INFO [main:CnxManagerTest@414] - Round {}, restarting peer {}[Ljava.lang.Object;@12bc6874
2016-07-12 22:34:59,157 [myid:] - INFO [main:FileSnap@83] - Reading snapshot /x1/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk8/branch-3.4/build/test/tmp/test8108741215991761367.junit.dir/version-2/snapshot.0
2016-07-12 22:34:59,159 [myid:] - INFO [ListenerThread:QuorumCnxManager$Listener@534] - My election bind port: /0.0.0.0:11241
2016-07-12 22:34:59,161 [myid:] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11240:QuorumPeer@774] - LOOKING
2016-07-12 22:34:59,162 [myid:] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11240:FastLeaderElection@818] - New election. My id = 0, proposed zxid=0x0
2016-07-12 22:34:59,162 [myid:] - INFO [WorkerReceiver[myid=0]:FastLeaderElection@600] - Notification: 1 (message format version), 0 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 0 (n.sid), 0x1 (n.peerEpoch) LOOKING (my state)
2016-07-12 22:34:59,163 [myid:] - INFO [WorkerSender[myid=0]:QuorumCnxManager@199] - Have smaller server identifier, so dropping the connection: (1, 0)
2016-07-12 22:34:59,163 [myid:] - INFO [/0.0.0.0:11244:QuorumCnxManager$Listener@541] - Received connection request /140.211.11.27:33142
2016-07-12 22:34:59,163 [myid:] - INFO [/0.0.0.0:11247:QuorumCnxManager$Listener@541] - Received connection request /140.211.11.27:44785
2016-07-12 22:34:59,163 [myid:] - INFO [WorkerSender[myid=0]:QuorumCnxManager@199] - Have smaller server identifier, so dropping the connection: (2, 0)
2016-07-12 22:34:59,164 [myid:] - INFO [/0.0.0.0:11241:QuorumCnxManager$Listener@541] - Received connection request /140.211.11.27:46884
2016-07-12 22:34:59,166 [myid:] - INFO [/0.0.0.0:11241:QuorumCnxManager$Listener@541] - Received connection request /140.211.11.27:46885
2016-07-12 22:34:59,167 [myid:] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message format version), 0 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 0 (n.sid), 0x1 (n.peerEpoch) FOLLOWING (my state)
2016-07-12 22:34:59,169 [myid:] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@600] - Notification: 1 (message format version), 0 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 0 (n.sid), 0x1 (n.peerEpoch) LEADING (my state)
2016-07-12 22:34:59,169 [myid:] - INFO [WorkerReceiver[myid=0]:FastLeaderElection@600] - Notification: 1 (message format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-12 22:34:59,169 [myid:] - INFO [WorkerReceiver[myid=0]:FastLeaderElection@600] - Notification: 1 (message format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-12 22:34:59,169 [myid:] - INFO [WorkerReceiver[myid=0]:FastLeaderElection@600] - Notification: 1 (message format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), FOLLOWING (n.state), 1 (n.sid), 0x1 (n.peerEpoch) LOOKING (my state)
2016-07-12 22:34:59,170 [myid:] - INFO [WorkerReceiver[myid=0]:FastLeaderElection@600] - Notification: 1 (message format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LEADING (n.state), 2 (n.sid), 0x1 (n.peerEpoch) LOOKING (my state)
2016-07-12 22:34:59,170 [myid:] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11240:QuorumPeer@844] - FOLLOWING
2016-07-12 22:34:59,170 [myid:] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11240:ZooKeeperServer@170] - Created server with tickTime 1000 minSessionTimeout 2000 maxSessionTimeout 20000 datadir /x1/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk8/branch-3.4/build/test/tmp/test8108741215991761367.junit.dir/version-2 snapdir /x1/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk8/branch-3.4/build/test/tmp/test8108741215991761367.junit.dir/version-2
2016-07-12 22:34:59,170 [myid:] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11240:Follower@63] - FOLLOWING - LEADER ELECTION TOOK - 8
2016-07-12 22:34:59,171 [myid:] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11240:QuorumPeer$QuorumServer@149] - Resolved hostname: 0.0.0.0 to address: /0.0.0.0
2016-07-12 22:34:59,172 [myid:] - INFO [LearnerHandler-/140.211.11.27:45228:LearnerHandler@329] - Follower sid: 0 : info : org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@23da3d
2016-07-12 22:34:59,172 [myid:] - INFO [LearnerHandler-/140.211.11.27:45228:LearnerHandler@384] - Synchronizing with Follower sid: 0 maxCommittedLog=0x0 minCommittedLog=0x0 peerLastZxid=0x0
2016-07-12 22:34:59,172 [myid:] - INFO [LearnerHandler-/140.211.11.27:45228:LearnerHandler@458] - Sending SNAP
2016-07-12 22:34:59,172 [myid:] - INFO [LearnerHandler-/140.211.11.27:45228:LearnerHandler@482] - Sending snapshot last zxid of peer is 0x0 zxid of leader is 0x100000000sent zxid of db as 0x100000000
2016-07-12 22:34:59,172 [myid:] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11240:Learner@329] - Getting a snapshot from leader
2016-07-12 22:34:59,174 [myid:] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11240:FileTxnSnapLog@240] - Snapshotting: 0x100000000 to /x1/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk8/branch-3.4/build/test/tmp/test8108741215991761367.junit.dir/version-2/snapshot.100000000
2016-07-12 22:34:59,284 [myid:] - INFO [LearnerHandler-/140.211.11.27:45228:LearnerHandler@518] - Received NEWLEADER-ACK message from 0
2016-07-12 22:34:59,652 [myid:] - INFO [/0.0.0.0:11241:QuorumCnxManager$Listener@560] - Leaving listener
2016-07-12 22:34:59,660 [myid:] - INFO [main:CnxManagerTest@405] - Round 1, halting peer 0
2016-07-12 22:34:59,660 [myid:] - INFO [main:Follower@166] - shutdown called
j
...[truncated 254755 chars]...
Keeper_branch34_jdk8/branch-3.4/build/test/tmp/test4996523116536588259.junit.dir/version-2 snapdir /x1/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk8/branch-3.4/build/test/tmp/test4996523116536588259.junit.dir/version-2
2016-07-12 22:35:13,328 [myid:] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11246:Follower@63] - FOLLOWING - LEADER ELECTION TOOK - 3
2016-07-12 22:35:13,329 [myid:] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11246:QuorumPeer$QuorumServer@149] - Resolved hostname: 0.0.0.0 to address: /0.0.0.0
2016-07-12 22:35:13,329 [myid:] - INFO [LearnerHandler-/140.211.11.27:32997:LearnerHandler@329] - Follower sid: 2 : info : org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@6f23fc6d
2016-07-12 22:35:13,330 [myid:] - INFO [LearnerHandler-/140.211.11.27:32997:LearnerHandler@384] - Synchronizing with Follower sid: 2 maxCommittedLog=0x0 minCommittedLog=0x0 peerLastZxid=0x200000000
2016-07-12 22:35:13,330 [myid:] - INFO [LearnerHandler-/140.211.11.27:32997:LearnerHandler@393] - leader and follower are in sync, zxid=0x200000000
2016-07-12 22:35:13,330 [myid:] - INFO [LearnerHandler-/140.211.11.27:32997:LearnerHandler@458] - Sending DIFF
2016-07-12 22:35:13,330 [myid:] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11246:Learner@326] - Getting a diff from the leader 0x200000000
2016-07-12 22:35:13,331 [myid:] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11246:FileTxnSnapLog@240] - Snapshotting: 0x200000000 to /x1/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk8/branch-3.4/build/test/tmp/test4996523116536588259.junit.dir/version-2/snapshot.200000000
2016-07-12 22:35:13,376 [myid:] - INFO [LearnerHandler-/140.211.11.27:32997:LearnerHandler@518] - Received NEWLEADER-ACK message from 2
2016-07-12 22:35:13,821 [myid:] - INFO [/0.0.0.0:11247:QuorumCnxManager$Listener@560] - Leaving listener
2016-07-12 22:35:13,824 [myid:] - INFO [main:Follower@166] - shutdown called
java.lang.Exception: shutdown Follower
at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:166)
at org.apache.zookeeper.server.quorum.QuorumPeer.shutdown(QuorumPeer.java:891)
at org.apache.zookeeper.server.quorum.CnxManagerTest.testWorkerThreads(CnxManagerTest.java:424)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:55)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
at org.junit.rules.TestWatchman$1.evaluate(TestWatchman.java:48)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:76)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184)
at org.junit.runners.ParentRunner.run(ParentRunner.java:236)
at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:39)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:518)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1052)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:906)
2016-07-12 22:35:13,824 [myid:] - INFO [main:FollowerZooKeeperServer@140] - Shutting down
2016-07-12 22:35:13,824 [myid:] - INFO [main:ZooKeeperServer@469] - shutting down
2016-07-12 22:35:13,825 [myid:] - INFO [main:FollowerRequestProcessor@107] - Shutting down
2016-07-12 22:35:13,825 [myid:] - INFO [main:CommitProcessor@184] - Shutting down
2016-07-12 22:35:13,825 [myid:] - INFO [main:FinalRequestProcessor@402] - shutdown of request processor complete
2016-07-12 22:35:13,825 [myid:] - INFO [CommitProcessor:0:CommitProcessor@153] - CommitProcessor exited loop!
2016-07-12 22:35:13,826 [myid:] - INFO [main:SyncRequestProcessor@209] - Shutting down
2016-07-12 22:35:13,825 [myid:] - INFO [FollowerRequestProcessor:0:FollowerRequestProcessor@97] - FollowerRequestProcessor exited loop!
2016-07-12 22:35:13,826 [myid:] - INFO [SyncThread:0:SyncRequestProcessor@187] - SyncRequestProcessor exited!
2016-07-12 22:35:13,827 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11240:NIOServerCnxnFactory@219] - NIOServerCnxn factory exited run method
2016-07-12 22:35:13,828 [myid:] - ERROR [/0.0.0.0:11241:QuorumCnxManager$Listener@547] - Exception while listening
java.net.SocketException: Socket closed
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:404)
at java.net.ServerSocket.implAccept(ServerSocket.java:545)
at java.net.ServerSocket.accept(ServerSocket.java:513)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$Listener.run(QuorumCnxManager.java:539)
2016-07-12 22:35:13,829 [myid:] - INFO [main:Leader@496] - Shutting down
2016-07-12 22:35:13,828 [myid:] - WARN [RecvWorker:0:QuorumCnxManager$RecvWorker@810] - Connection broken for id 0, my id = 1, error =
java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:392)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:795)
2016-07-12 22:35:13,829 [myid:] - WARN [RecvWorker:0:QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker
2016-07-12 22:35:13,828 [myid:] - WARN [RecvWorker:0:QuorumCnxManager$RecvWorker@810] - Connection broken for id 0, my id = 2, error =
java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:392)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:795)
2016-07-12 22:35:13,829 [myid:] - WARN [RecvWorker:0:QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker
2016-07-12 22:35:13,828 [myid:] - WARN [SendWorker:2:QuorumCnxManager$SendWorker@727] - Interrupted while waiting for message on queue
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:879)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:65)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:715)
2016-07-12 22:35:13,830 [myid:] - WARN [SendWorker:2:QuorumCnxManager$SendWorker@736] - Send worker leaving thread
2016-07-12 22:35:13,828 [myid:] - WARN [SendWorker:1:QuorumCnxManager$SendWorker@727] - Interrupted while waiting for message on queue
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:879)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:65)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:715)
2016-07-12 22:35:13,830 [myid:] - WARN [SendWorker:1:QuorumCnxManager$SendWorker@736] - Send worker leaving thread
2016-07-12 22:35:13,828 [myid:] - WARN [RecvWorker:1:QuorumCnxManager$RecvWorker@810] - Connection broken for id 1, my id = 0, error =
java.net.SocketException: Socket closed
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:150)
at java.net.SocketInputStream.read(SocketInputStream.java:121)
at java.net.SocketInputStream.read(SocketInputStream.java:203)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:795)
2016-07-12 22:35:13,830 [myid:] - WARN [RecvWorker:1:QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker
2016-07-12 22:35:13,830 [myid:] - WARN [SendWorker:0:QuorumCnxManager$SendWorker@727] - Interrupted while waiting for message on queue
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:879)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:65)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:715)
2016-07-12 22:35:13,830 [myid:] - WARN [SendWorker:0:QuorumCnxManager$SendWorker@736] - Send worker leaving thread
2016-07-12 22:35:13,829 [myid:] - WARN [SendWorker:0:QuorumCnxManager$SendWorker@727] - Interrupted while waiting for message on queue
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:879)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:65)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:715)
2016-07-12 22:35:13,831 [myid:] - WARN [SendWorker:0:QuorumCnxManager$SendWorker@736] - Send worker leaving thread
2016-07-12 22:35:13,829 [myid:] - WARN [RecvWorker:2:QuorumCnxManager$RecvWorker@810] - Connection broken for id 2, my id = 0, error =
java.net.SocketException: Socket closed
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:150)
at java.net.SocketInputStream.read(SocketInputStream.java:121)
at java.net.SocketInputStream.read(SocketInputStream.java:203)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:795)
2016-07-12 22:35:13,831 [myid:] - WARN [RecvWorker:2:QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker
2016-07-12 22:35:13,829 [myid:] - INFO [main:Leader@502] - Shutdown called
java.lang.Exception: shutdown Leader! reason: quorum Peer shutdown
at org.apache.zookeeper.server.quorum.Leader.shutdown(Leader.java:502)
at org.apache.zookeeper.server.quorum.QuorumPeer.shutdown(QuorumPeer.java:888)
at org.apache.zookeeper.server.quorum.CnxManagerTest.testWorkerThreads(CnxManagerTest.java:424)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:55)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
at org.junit.rules.TestWatchman$1.evaluate(TestWatchman.java:48)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:76)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184)
at org.junit.runners.ParentRunner.run(ParentRunner.java:236)
at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:39)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:518)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1052)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:906)
2016-07-12 22:35:13,831 [myid:] - INFO [main:ZooKeeperServer@469] - shutting down
2016-07-12 22:35:13,831 [myid:] - INFO [main:SessionTrackerImpl@225] - Shutting down
2016-07-12 22:35:13,832 [myid:] - INFO [main:PrepRequestProcessor@765] - Shutting down
2016-07-12 22:35:13,832 [myid:] - INFO [main:ProposalRequestProcessor@88] - Shutting down
2016-07-12 22:35:13,832 [myid:] - INFO [main:CommitProcessor@184] - Shutting down
2016-07-12 22:35:13,832 [myid:] - INFO [main:Leader$ToBeAppliedRequestProcessor@661] - Shutting down
2016-07-12 22:35:13,832 [myid:] - INFO [main:FinalRequestProcessor@402] - shutdown of request processor complete
2016-07-12 22:35:13,832 [myid:] - INFO [main:SyncRequestProcessor@209] - Shutting down
2016-07-12 22:35:13,831 [myid:] - INFO [LearnerCnxAcceptor-/0.0.0.0:11242:Leader$LearnerCnxAcceptor@325] - exception while shutting down acceptor: java.net.SocketException: Socket closed
2016-07-12 22:35:13,832 [myid:] - INFO [SyncThread:1:SyncRequestProcessor@187] - SyncRequestProcessor exited!
2016-07-12 22:35:13,832 [myid:] - INFO [CommitProcessor:1:CommitProcessor@153] - CommitProcessor exited loop!
2016-07-12 22:35:13,832 [myid:] - INFO [ProcessThread(sid:1 cport:-1)::PrepRequestProcessor@143] - PrepRequestProcessor exited loop!
2016-07-12 22:35:13,834 [myid:] - WARN [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11246:Follower@89] - Exception when following the leader
java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:392)
at org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63)
at org.apache.zookeeper.server.quorum.QuorumPacket.deserialize(QuorumPacket.java:83)
at org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:99)
at org.apache.zookeeper.server.quorum.Learner.readPacket(Learner.java:153)
at org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:85)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:846)
2016-07-12 22:35:13,833 [myid:] - WARN [LearnerHandler-/140.211.11.27:32993:LearnerHandler@644] - ******* GOODBYE /140.211.11.27:32993 ********
2016-07-12 22:35:13,835 [myid:] - WARN [LearnerHandler-/140.211.11.27:32990:LearnerHandler@644] - ******* GOODBYE /140.211.11.27:32990 ********
2016-07-12 22:35:13,834 [myid:] - WARN [LearnerHandler-/140.211.11.27:32980:LearnerHandler@644] - ******* GOODBYE /140.211.11.27:32980 ********
2016-07-12 22:35:13,835 [myid:] - WARN [LearnerHandler-/140.211.11.27:32980:LearnerHandler@656] - Ignoring unexpected exception
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1220)
at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:335)
at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:339)
at org.apache.zookeeper.server.quorum.LearnerHandler.shutdown(LearnerHandler.java:654)
at org.apache.zookeeper.server.quorum.LearnerHandler.run(LearnerHandler.java:647)
2016-07-12 22:35:13,834 [myid:] - WARN [LearnerHandler-/140.211.11.27:32997:LearnerHandler@644] - ******* GOODBYE /140.211.11.27:32997 ********
2016-07-12 22:35:13,835 [myid:] - WARN [LearnerHandler-/140.211.11.27:32997:LearnerHandler@656] - Ignoring unexpected exception
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1220)
at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:335)
at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:339)
at org.apache.zookeeper.server.quorum.LearnerHandler.shutdown(LearnerHandler.java:654)
at org.apache.zookeeper.server.quorum.LearnerHandler.run(LearnerHandler.java:647)
2016-07-12 22:35:13,835 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11243:NIOServerCnxnFactory@219] - NIOServerCnxn factory exited run method
2016-07-12 22:35:13,835 [myid:] - WARN [LearnerHandler-/140.211.11.27:32990:LearnerHandler@656] - Ignoring unexpected exception
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1220)
at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:335)
at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:339)
at org.apache.zookeeper.server.quorum.LearnerHandler.shutdown(LearnerHandler.java:654)
at org.apache.zookeeper.server.quorum.LearnerHandler.run(LearnerHandler.java:647)
2016-07-12 22:35:13,836 [myid:] - WARN [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11243:QuorumPeer@862] - Unexpected exception
java.lang.InterruptedException: sleep interrupted
at java.lang.Thread.sleep(Native Method)
at org.apache.zookeeper.server.quorum.Leader.lead(Leader.java:456)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:859)
2016-07-12 22:35:13,836 [myid:] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11243:Leader@496] - Shutting down
2016-07-12 22:35:13,836 [myid:] - WARN [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11243:QuorumPeer@874] - QuorumPeer main thread exited
2016-07-12 22:35:13,835 [myid:] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11246:Follower@166] - shutdown called
java.lang.Exception: shutdown Follower
at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:166)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:850)
2016-07-12 22:35:13,837 [myid:] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11246:FollowerZooKeeperServer@140] - Shutting down
2016-07-12 22:35:13,837 [myid:] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11246:ZooKeeperServer@469] - shutting down
2016-07-12 22:35:13,837 [myid:] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11246:FollowerRequestProcessor@107] - Shutting down
2016-07-12 22:35:13,835 [myid:] - WARN [LearnerHandler-/140.211.11.27:32993:LearnerHandler@656] - Ignoring unexpected exception
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1220)
at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:335)
at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:339)
at org.apache.zookeeper.server.quorum.LearnerHandler.shutdown(LearnerHandler.java:654)
at org.apache.zookeeper.server.quorum.LearnerHandler.run(LearnerHandler.java:647)
2016-07-12 22:35:13,835 [myid:] - WARN [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11240:Follower@89] - Exception when following the leader
java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:392)
at org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63)
at org.apache.zookeeper.server.quorum.QuorumPacket.deserialize(QuorumPacket.java:83)
at org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:99)
at org.apache.zookeeper.server.quorum.Learner.readPacket(Learner.java:153)
at org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:85)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:846)
2016-07-12 22:35:13,837 [myid:] - INFO [FollowerRequestProcessor:2:FollowerRequestProcessor@97] - FollowerRequestProcessor exited loop!
2016-07-12 22:35:13,838 [myid:] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11240:Follower@166] - shutdown called
java.lang.Exception: shutdown Follower
at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:166)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:850)
2016-07-12 22:35:13,837 [myid:] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11246:CommitProcessor@184] - Shutting down
2016-07-12 22:35:13,839 [myid:] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11246:FinalRequestProcessor@402] - shutdown of request processor complete
2016-07-12 22:35:13,839 [myid:] - INFO [CommitProcessor:2:CommitProcessor@153] - CommitProcessor exited loop!
2016-07-12 22:35:13,837 [myid:] - INFO [main:Follower@166] - shutdown called
java.lang.Exception: shutdown Follower
at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:166)
at org.apache.zookeeper.server.quorum.QuorumPeer.shutdown(QuorumPeer.java:891)
at org.apache.zookeeper.server.quorum.CnxManagerTest.testWorkerThreads(CnxManagerTest.java:424)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:55)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
at org.junit.rules.TestWatchman$1.evaluate(TestWatchman.java:48)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:76)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184)
at org.junit.runners.ParentRunner.run(ParentRunner.java:236)
at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:39)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:518)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1052)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:906)
2016-07-12 22:35:13,839 [myid:] - INFO [main:FollowerZooKeeperServer@140] - Shutting down
2016-07-12 22:35:13,839 [myid:] - INFO [main:SyncRequestProcessor@209] - Shutting down
2016-07-12 22:35:13,836 [myid:] - WARN [RecvWorker:2:QuorumCnxManager$RecvWorker@810] - Connection broken for id 2, my id = 1, error =
java.net.SocketException: Socket closed
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:150)
at java.net.SocketInputStream.read(SocketInputStream.java:121)
at java.net.SocketInputStream.read(SocketInputStream.java:203)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:795)
2016-07-12 22:35:13,840 [myid:] - WARN [RecvWorker:2:QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker
2016-07-12 22:35:13,836 [myid:] - WARN [RecvWorker:1:QuorumCnxManager$RecvWorker@810] - Connection broken for id 1, my id = 2, error =
java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:392)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:795)
2016-07-12 22:35:13,840 [myid:] - WARN [RecvWorker:1:QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker
2016-07-12 22:35:13,836 [myid:] - WARN [SendWorker:2:QuorumCnxManager$SendWorker@727] - Interrupted while waiting for message on queue
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:879)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:65)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:715)
2016-07-12 22:35:13,840 [myid:] - WARN [SendWorker:2:QuorumCnxManager$SendWorker@736] - Send worker leaving thread
2016-07-12 22:35:13,836 [myid:] - ERROR [/0.0.0.0:11244:QuorumCnxManager$Listener@547] - Exception while listening
java.net.SocketException: Socket closed
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:404)
at java.net.ServerSocket.implAccept(ServerSocket.java:545)
at java.net.ServerSocket.accept(ServerSocket.java:513)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$Listener.run(QuorumCnxManager.java:539)
2016-07-12 22:35:13,840 [myid:] - WARN [SendWorker:1:QuorumCnxManager$SendWorker@727] - Interrupted while waiting for message on queue
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:879)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:65)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:715)
2016-07-12 22:35:13,841 [myid:] - WARN [SendWorker:1:QuorumCnxManager$SendWorker@736] - Send worker leaving thread
2016-07-12 22:35:13,840 [myid:] - INFO [SyncThread:2:SyncRequestProcessor@187] - SyncRequestProcessor exited!
2016-07-12 22:35:13,839 [myid:] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11246:SyncRequestProcessor@209] - Shutting down
2016-07-12 22:35:13,841 [myid:] - WARN [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11246:QuorumPeer@874] - QuorumPeer main thread exited
2016-07-12 22:35:13,841 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11246:NIOServerCnxnFactory@219] - NIOServerCnxn factory exited run method
2016-07-12 22:35:13,839 [myid:] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11240:FollowerZooKeeperServer@140] - Shutting down
2016-07-12 22:35:13,842 [myid:] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11240:SyncRequestProcessor@209] - Shutting down
2016-07-12 22:35:13,842 [myid:] - WARN [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:11240:QuorumPeer@874] - QuorumPeer main thread exited
2016-07-12 22:35:13,842 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@58] - Memory used 11673
2016-07-12 22:35:13,842 [myid:] - ERROR [/0.0.0.0:11247:QuorumCnxManager$Listener@547] - Exception while listening
java.net.SocketException: Socket closed
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:404)
at java.net.ServerSocket.implAccept(ServerSocket.java:545)
at java.net.ServerSocket.accept(ServerSocket.java:513)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$Listener.run(QuorumCnxManager.java:539)
2016-07-12 22:35:13,842 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@63] - Number of threads 20
2016-07-12 22:35:13,842 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@78] - FINISHED TEST METHOD testWorkerThreads
2016-07-12 22:35:13,842 [myid:] - INFO [main:ZKTestCase$1@60] - SUCCEEDED testWorkerThreads
2016-07-12 22:35:13,843 [myid:] - INFO [main:ZKTestCase$1@55] - FINISHED testWorkerThreads
2016-07-12 22:35:13,843 [myid:] - INFO [main:ZKTestCase$1@50] - STARTING testCnxManager
2016-07-12 22:35:13,843 [myid:] - INFO [main:PortAssignment@32] - assigning port 11248
2016-07-12 22:35:13,843 [myid:] - INFO [main:PortAssignment@32] - assigning port 11249
2016-07-12 22:35:13,843 [myid:] - INFO [main:PortAssignment@32] - assigning port 11250
2016-07-12 22:35:13,843 [myid:] - INFO [main:QuorumPeer$QuorumServer@149] - Resolved hostname: 0.0.0.0 to address: /0.0.0.0
2016-07-12 22:35:13,844 [myid:] - INFO [main:PortAssignment@32] - assigning port 11251
2016-07-12 22:35:13,844 [myid:] - INFO [main:PortAssignment@32] - assigning port 11252
2016-07-12 22:35:13,844 [myid:] - INFO [main:PortAssignment@32] - assigning port 11253
2016-07-12 22:35:13,844 [myid:] - INFO [main:QuorumPeer$QuorumServer@149] - Resolved hostname: 0.0.0.0 to address: /0.0.0.0
2016-07-12 22:35:13,844 [myid:] - INFO [main:PortAssignment@32] - assigning port 11254
2016-07-12 22:35:13,844 [myid:] - INFO [main:PortAssignment@32] - assigning port 11255
2016-07-12 22:35:13,844 [myid:] - INFO [main:PortAssignment@32] - assigning port 11256
2016-07-12 22:35:13,844 [myid:] - INFO [main:QuorumPeer$QuorumServer@149] - Resolved hostname: 0.0.0.0 to address: /0.0.0.0
2016-07-12 22:35:13,844 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@53] - RUNNING TEST METHOD testCnxManager
2016-07-12 22:35:13,845 [myid:] - INFO [main:NIOServerCnxnFactory@89] - binding to port 0.0.0.0/0.0.0.0:11252
2016-07-12 22:35:13,846 [myid:] - INFO [Thread-19:NIOServerCnxnFactory@89] - binding to port 0.0.0.0/0.0.0.0:11249
2016-07-12 22:35:13,846 [myid:] - INFO [ListenerThread:QuorumCnxManager$Listener@534] - My election bind port: /0.0.0.0:11253
2016-07-12 22:35:13,846 [myid:] - WARN [main:QuorumCnxManager@400] - Cannot open channel to 0 at election address /0.0.0.0:11250
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:345)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:204)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:381)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:354)
at org.apache.zookeeper.server.quorum.CnxManagerTest.testCnxManager(CnxManagerTest.java:142)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:55)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
at org.junit.rules.TestWatchman$1.evaluate(TestWatchman.java:48)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:76)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184)
at org.junit.runners.ParentRunner.run(ParentRunner.java:236)
at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:39)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:518)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1052)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:906)
2016-07-12 22:35:13,847 [myid:] - INFO [main:QuorumPeer$QuorumServer@149] - Resolved hostname: 0.0.0.0 to address: /0.0.0.0
2016-07-12 22:35:13,847 [myid:] - INFO [ListenerThread:QuorumCnxManager$Listener@534] - My election bind port: /0.0.0.0:11250
2016-07-12 22:35:13,847 [myid:] - INFO [/0.0.0.0:11253:QuorumCnxManager$Listener@541] - Received connection request /140.211.11.27:41746
2016-07-12 22:35:13,847 [myid:] - INFO [Thread-19:QuorumCnxManager@199] - Have smaller server identifier, so dropping the connection: (1, 0)
2016-07-12 22:35:13,848 [myid:] - INFO [/0.0.0.0:11250:QuorumCnxManager$Listener@541] - Received connection request /140.211.11.27:46883
2016-07-12 22:35:13,849 [myid:] - INFO [/0.0.0.0:11253:QuorumCnxManager$Listener@541] - Received connection request /140.211.11.27:41748
2016-07-12 22:35:13,849 [myid:] - INFO [Thread-19:QuorumCnxManager@199] - Have smaller server identifier, so dropping the connection: (1, 0)
2016-07-12 22:35:13,849 [myid:] - WARN [SendWorker:0:QuorumCnxManager$SendWorker@727] - Interrupted while waiting for message on queue
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:879)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:65)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:715)
2016-07-12 22:35:13,849 [myid:] - INFO [/0.0.0.0:11250:QuorumCnxManager$Listener@541] - Received connection request /140.211.11.27:46885
2016-07-12 22:35:13,849 [myid:] - WARN [SendWorker:0:QuorumCnxManager$SendWorker@736] - Send worker leaving thread
2016-07-12 22:35:13,849 [myid:] - WARN [RecvWorker:0:QuorumCnxManager$RecvWorker@810] - Connection broken for id 0, my id = 1, error =
java.net.SocketException: Socket closed
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:150)
at java.net.SocketInputStream.read(SocketInputStream.java:121)
at java.net.SocketInputStream.read(SocketInputStream.java:203)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:795)
2016-07-12 22:35:13,850 [myid:] - WARN [RecvWorker:0:QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker
2016-07-12 22:35:13,849 [myid:] - WARN [RecvWorker:1:QuorumCnxManager$RecvWorker@810] - Connection broken for id 1, my id = 0, error =
java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:392)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:795)
2016-07-12 22:35:13,850 [myid:] - WARN [RecvWorker:1:QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker
2016-07-12 22:35:13,850 [myid:] - WARN [SendWorker:1:QuorumCnxManager$SendWorker@727] - Interrupted while waiting for message on queue
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:879)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:65)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:715)
2016-07-12 22:35:13,851 [myid:] - WARN [SendWorker:1:QuorumCnxManager$SendWorker@736] - Send worker leaving thread
2016-07-12 22:35:13,851 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@58] - Memory used 12332
2016-07-12 22:35:13,851 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@63] - Number of threads 26
2016-07-12 22:35:13,851 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@78] - FINISHED TEST METHOD testCnxManager
2016-07-12 22:35:13,851 [myid:] - INFO [main:ZKTestCase$1@60] - SUCCEEDED testCnxManager
2016-07-12 22:35:13,851 [myid:] - INFO [main:ZKTestCase$1@55] - FINISHED testCnxManager
2016-07-12 22:35:13,862 [myid:] - INFO [main:ZKTestCase$1@50] - STARTING testCnxManagerTimeout
2016-07-12 22:35:13,862 [myid:] - INFO [main:PortAssignment@32] - assigning port 11257
2016-07-12 22:35:13,862 [myid:] - INFO [main:PortAssignment@32] - assigning port 11258
2016-07-12 22:35:13,862 [myid:] - INFO [main:PortAssignment@32] - assigning port 11259
2016-07-12 22:35:13,862 [myid:] - INFO [main:QuorumPeer$QuorumServer@149] - Resolved hostname: 0.0.0.0 to address: /0.0.0.0
2016-07-12 22:35:13,862 [myid:] - INFO [main:PortAssignment@32] - assigning port 11260
2016-07-12 22:35:13,862 [myid:] - INFO [main:PortAssignment@32] - assigning port 11261
2016-07-12 22:35:13,862 [myid:] - INFO [main:PortAssignment@32] - assigning port 11262
2016-07-12 22:35:13,862 [myid:] - INFO [main:QuorumPeer$QuorumServer@149] - Resolved hostname: 0.0.0.0 to address: /0.0.0.0
2016-07-12 22:35:13,863 [myid:] - INFO [main:PortAssignment@32] - assigning port 11263
2016-07-12 22:35:13,863 [myid:] - INFO [main:PortAssignment@32] - assigning port 11264
2016-07-12 22:35:13,863 [myid:] - INFO [main:PortAssignment@32] - assigning port 11265
2016-07-12 22:35:13,863 [myid:] - INFO [main:QuorumPeer$QuorumServer@149] - Resolved hostname: 0.0.0.0 to address: /0.0.0.0
2016-07-12 22:35:13,863 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@53] - RUNNING TEST METHOD testCnxManagerTimeout
2016-07-12 22:35:13,863 [myid:] - INFO [main:PortAssignment@32] - assigning port 11266
2016-07-12 22:35:13,863 [myid:] - INFO [main:CnxManagerTest@171] - This is the dead address I'm trying: 192.0.2.179
2016-07-12 22:35:13,863 [myid:] - INFO [main:PortAssignment@32] - assigning port 11267
2016-07-12 22:35:13,863 [myid:] - INFO [main:QuorumPeer$QuorumServer@149] - Resolved hostname: 192.0.2.179 to address: /192.0.2.179
2016-07-12 22:35:13,864 [myid:] - INFO [main:NIOServerCnxnFactory@89] - binding to port 0.0.0.0/0.0.0.0:11261
2016-07-12 22:35:13,864 [myid:] - INFO [ListenerThread:QuorumCnxManager$Listener@534] - My election bind port: /0.0.0.0:11262
2016-07-12 22:35:14,000 [myid:] - INFO [SessionTracker:SessionTrackerImpl@162] - SessionTrackerImpl exited loop!
2016-07-12 22:35:14,310 [myid:] - INFO [WorkerSender[myid=2]:FastLeaderElection$Messenger$WorkerSender@438] - WorkerSender is down
2016-07-12 22:35:14,311 [myid:] - INFO [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@407] - WorkerReceiver is down
2016-07-12 22:35:14,829 [myid:] - INFO [/0.0.0.0:11241:QuorumCnxManager$Listener@560] - Leaving listener
2016-07-12 22:35:14,840 [myid:] - INFO [/0.0.0.0:11244:QuorumCnxManager$Listener@560] - Leaving listener
2016-07-12 22:35:14,842 [myid:] - INFO [/0.0.0.0:11247:QuorumCnxManager$Listener@560] - Leaving listener
2016-07-12 22:35:15,319 [myid:] - INFO [WorkerSender[myid=2]:FastLeaderElection$Messenger$WorkerSender@438] - WorkerSender is down
2016-07-12 22:35:15,320 [myid:] - INFO [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@407] - WorkerReceiver is down
2016-07-12 22:35:16,327 [myid:] - INFO [WorkerReceiver[myid=0]:FastLeaderElection$Messenger$WorkerReceiver@407] - WorkerReceiver is down
2016-07-12 22:35:16,327 [myid:] - INFO [WorkerSender[myid=2]:FastLeaderElection$Messenger$WorkerSender@438] - WorkerSender is down
2016-07-12 22:35:16,327 [myid:] - INFO [WorkerSender[myid=0]:FastLeaderElection$Messenger$WorkerSender@438] - WorkerSender is down
2016-07-12 22:35:16,328 [myid:] - INFO [WorkerReceiver[myid=1]:FastLeaderElection$Messenger$WorkerReceiver@407] - WorkerReceiver is down
2016-07-12 22:35:16,328 [myid:] - INFO [WorkerSender[myid=1]:FastLeaderElection$Messenger$WorkerSender@438] - WorkerSender is down
2016-07-12 22:35:16,328 [myid:] - INFO [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@407] - WorkerReceiver is down
2016-07-12 22:35:18,869 [myid:] - WARN [main:QuorumCnxManager@400] - Cannot open channel to 2 at election address /192.0.2.179:11267
java.net.SocketTimeoutException: connect timed out
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:345)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:381)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:354)
at org.apache.zookeeper.server.quorum.CnxManagerTest.testCnxManagerTimeout(CnxManagerTest.java:187)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:55)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
at org.junit.rules.TestWatchman$1.evaluate(TestWatchman.java:48)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:76)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184)
at org.junit.runners.ParentRunner.run(ParentRunner.java:236)
at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:39)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:518)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1052)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:906)
2016-07-12 22:35:18,870 [myid:] - INFO [main:QuorumPeer$QuorumServer@149] - Resolved hostname: 192.0.2.179 to address: /192.0.2.179
2016-07-12 22:35:18,870 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@58] - Memory used 12750
2016-07-12 22:35:18,870 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@63] - Number of threads 13
2016-07-12 22:35:18,871 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@78] - FINISHED TEST METHOD testCnxManagerTimeout
2016-07-12 22:35:18,871 [myid:] - INFO [main:ZKTestCase$1@60] - SUCCEEDED testCnxManagerTimeout
2016-07-12 22:35:18,871 [myid:] - INFO [main:ZKTestCase$1@55] - FINISHED testCnxManagerTimeout
2016-07-12 22:35:18,871 [myid:] - INFO [main:ZKTestCase$1@50] - STARTING testCnxManagerSpinLock
2016-07-12 22:35:18,872 [myid:] - INFO [main:PortAssignment@32] - assigning port 11268
2016-07-12 22:35:18,872 [myid:] - INFO [main:PortAssignment@32] - assigning port 11269
2016-07-12 22:35:18,872 [myid:] - INFO [main:PortAssignment@32] - assigning port 11270
2016-07-12 22:35:18,872 [myid:] - INFO [main:QuorumPeer$QuorumServer@149] - Resolved hostname: 0.0.0.0 to address: /0.0.0.0
2016-07-12 22:35:18,872 [myid:] - INFO [main:PortAssignment@32] - assigning port 11271
2016-07-12 22:35:18,872 [myid:] - INFO [main:PortAssignment@32] - assigning port 11272
2016-07-12 22:35:18,872 [myid:] - INFO [main:PortAssignment@32] - assigning port 11273
2016-07-12 22:35:18,873 [myid:] - INFO [main:QuorumPeer$QuorumServer@149] - Resolved hostname: 0.0.0.0 to address: /0.0.0.0
2016-07-12 22:35:18,873 [myid:] - INFO [main:PortAssignment@32] - assigning port 11274
2016-07-12 22:35:18,873 [myid:] - INFO [main:PortAssignment@32] - assigning port 11275
2016-07-12 22:35:18,873 [myid:] - INFO [main:PortAssignment@32] - assigning port 11276
2016-07-12 22:35:18,873 [myid:] - INFO [main:QuorumPeer$QuorumServer@149] - Resolved hostname: 0.0.0.0 to address: /0.0.0.0
2016-07-12 22:35:18,874 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@53] - RUNNING TEST METHOD testCnxManagerSpinLock
2016-07-12 22:35:18,874 [myid:] - INFO [main:NIOServerCnxnFactory@89] - binding to port 0.0.0.0/0.0.0.0:11272
2016-07-12 22:35:18,875 [myid:] - INFO [main:CnxManagerTest@214] - Election port: 11273
2016-07-12 22:35:18,875 [myid:] - INFO [ListenerThread:QuorumCnxManager$Listener@534] - My election bind port: /0.0.0.0:11273
2016-07-12 22:35:19,876 [myid:] - INFO [/0.0.0.0:11273:QuorumCnxManager$Listener@541] - Received connection request /140.211.11.27:37939
2016-07-12 22:35:19,876 [myid:] - WARN [RecvWorker:2:QuorumCnxManager$RecvWorker@810] - Connection broken for id 2, my id = 1, error =
java.io.IOException: Received packet with invalid packet: -20
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:797)
2016-07-12 22:35:19,877 [myid:] - WARN [RecvWorker:2:QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker
2016-07-12 22:35:19,877 [myid:] - WARN [SendWorker:2:QuorumCnxManager$SendWorker@727] - Interrupted while waiting for message on queue
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:879)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:65)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:715)
2016-07-12 22:35:19,877 [myid:] - WARN [SendWorker:2:QuorumCnxManager$SendWorker@736] - Send worker leaving thread
2016-07-12 22:35:20,876 [myid:] - INFO [main:CnxManagerTest@249] - Socket has been closed as expected
2016-07-12 22:35:20,876 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@58] - Memory used 12944
2016-07-12 22:35:20,876 [myid:] - ERROR [/0.0.0.0:11273:QuorumCnxManager$Listener@547] - Exception while listening
java.net.SocketException: Socket closed
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:404)
at java.net.ServerSocket.implAccept(ServerSocket.java:545)
at java.net.ServerSocket.accept(ServerSocket.java:513)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$Listener.run(QuorumCnxManager.java:539)
2016-07-12 22:35:20,877 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@63] - Number of threads 14
2016-07-12 22:35:20,877 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@78] - FINISHED TEST METHOD testCnxManagerSpinLock
2016-07-12 22:35:20,877 [myid:] - INFO [main:ZKTestCase$1@60] - SUCCEEDED testCnxManagerSpinLock
2016-07-12 22:35:20,877 [myid:] - INFO [main:ZKTestCase$1@55] - FINISHED testCnxManagerSpinLock
{noformat}
flaky, flaky-test 9223372036854775807 No Perforce job exists for this issue. 2 9223372036854775807
3 years, 28 weeks, 2 days ago
Reviewed
0|i31yvj:
ZooKeeper ZOOKEEPER-2501

Improve ZooKeeperServerListenerImpl#notifyStopping() logging

Improvement Open Minor Unresolved Rakesh Radhakrishnan Rakesh Radhakrishnan Rakesh Radhakrishnan 04/Aug/16 14:43   04/Aug/16 15:03   3.4.8, 3.5.2   server   0 1   This jira is to address [~fpj]'s comment.

{code}
ZooKeeperCriticalThread#handleException() logs an error message including a throwable. Again, in ZooKeeperServerListenerImpl#notifyStopping(), it logs an info message with the exit code. Here it is better to consolidate the logging and have only a log error message here in this #notifyStopping method. We would need to change the signature and pass a throwable, though.
{code}
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 33 weeks ago 0|i31xdj:
ZooKeeper ZOOKEEPER-2500

Fix compilation warnings for CliException classes

Bug Closed Major Fixed Abraham Fine Abraham Fine Abraham Fine 03/Aug/16 11:39   17/May/17 23:44 05/Aug/16 17:07 3.5.3 3.5.3, 3.6.0     0 3   9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
3 years, 32 weeks, 6 days ago
Reviewed
0|i31uzr:
ZooKeeper ZOOKEEPER-2499

ZOOKEEPER-3170 Flaky Test: org.apache.zookeeper.test.SSLTest.testSecureQuorumServer

Sub-task Closed Major Cannot Reproduce Andor Molnar Michael Han Michael Han 03/Aug/16 01:44   19/Dec/19 18:01 25/Oct/18 11:20 3.5.2 3.5.5 tests   1 3  
{noformat}
Error Message

waiting for server 0 being up
Stacktrace

junit.framework.AssertionFailedError: waiting for server 0 being up
at org.apache.zookeeper.test.SSLTest.testSecureQuorumServer(SSLTest.java:100)
at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:79)
Standard Output

2016-08-03 05:33:41,529 [myid:] - INFO [main:JUnit4ZKTestRunner@47] - No test.method specified. using default methods.
2016-08-03 05:33:41,594 [myid:] - INFO [main:JUnit4ZKTestRunner@47] - No test.method specified. using default methods.
2016-08-03 05:33:41,608 [myid:] - INFO [main:ZKTestCase$1@55] - STARTING testSecureQuorumServer
2016-08-03 05:33:41,611 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@77] - RUNNING TEST METHOD testSecureQuorumServer
2016-08-03 05:33:41,614 [myid:] - INFO [main:PortAssignment@151] - Test process 8/8 using ports from 30072 - 32764.
2016-08-03 05:33:41,616 [myid:] - INFO [main:PortAssignment@85] - Assigned port 30073 from range 30072 - 32764.
2016-08-03 05:33:41,617 [myid:] - INFO [main:PortAssignment@85] - Assigned port 30074 from range 30072 - 32764.
2016-08-03 05:33:41,617 [myid:] - INFO [main:PortAssignment@85] - Assigned port 30075 from range 30072 - 32764.
2016-08-03 05:33:41,617 [myid:] - INFO [main:PortAssignment@85] - Assigned port 30076 from range 30072 - 32764.
2016-08-03 05:33:41,618 [myid:] - INFO [main:PortAssignment@85] - Assigned port 30077 from range 30072 - 32764.
2016-08-03 05:33:41,618 [myid:] - INFO [main:PortAssignment@85] - Assigned port 30078 from range 30072 - 32764.
2016-08-03 05:33:41,618 [myid:] - INFO [main:PortAssignment@85] - Assigned port 30079 from range 30072 - 32764.
2016-08-03 05:33:41,619 [myid:] - INFO [main:PortAssignment@85] - Assigned port 30080 from range 30072 - 32764.
2016-08-03 05:33:41,619 [myid:] - INFO [main:PortAssignment@85] - Assigned port 30081 from range 30072 - 32764.
2016-08-03 05:33:41,620 [myid:] - INFO [main:PortAssignment@85] - Assigned port 30082 from range 30072 - 32764.
2016-08-03 05:33:41,620 [myid:] - INFO [main:PortAssignment@85] - Assigned port 30083 from range 30072 - 32764.
2016-08-03 05:33:41,623 [myid:] - INFO [main:PortAssignment@85] - Assigned port 30084 from range 30072 - 32764.
2016-08-03 05:33:41,641 [myid:] - INFO [main:QuorumPeerTestBase$MainThread@131] - id = 0 tmpDir = /home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/trunk/build/test/tmp/test8166855371662584608.junit.dir clientPort = -1 adminServerPort = 8080
2016-08-03 05:33:41,647 [myid:] - INFO [main:QuorumPeerTestBase$MainThread@131] - id = 1 tmpDir = /home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/trunk/build/test/tmp/test5311610672905591522.junit.dir clientPort = -1 adminServerPort = 8080
2016-08-03 05:33:41,648 [myid:] - INFO [main:QuorumPeerTestBase$MainThread@131] - id = 2 tmpDir = /home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/trunk/build/test/tmp/test4754133370740902988.junit.dir clientPort = -1 adminServerPort = 8080
2016-08-03 05:33:41,651 [myid:] - INFO [Thread-1:QuorumPeerConfig@116] - Reading configuration from: /home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/trunk/build/test/tmp/test5311610672905591522.junit.dir/zoo.cfg
2016-08-03 05:33:41,651 [myid:] - INFO [Thread-0:QuorumPeerConfig@116] - Reading configuration from: /home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/trunk/build/test/tmp/test8166855371662584608.junit.dir/zoo.cfg
2016-08-03 05:33:41,651 [myid:] - INFO [Thread-2:QuorumPeerConfig@116] - Reading configuration from: /home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/trunk/build/test/tmp/test4754133370740902988.junit.dir/zoo.cfg
2016-08-03 05:33:41,651 [myid:] - INFO [main:FourLetterWordMain@85] - connecting to 127.0.0.1 30073
2016-08-03 05:33:41,653 [myid:] - INFO [Thread-2:QuorumPeerConfig@308] - clientPort is not set
2016-08-03 05:33:41,653 [myid:] - INFO [Thread-1:QuorumPeerConfig@308] - clientPort is not set
2016-08-03 05:33:41,653 [myid:] - INFO [Thread-1:QuorumPeerConfig@332] - secureClientPortAddress is 0.0.0.0/0.0.0.0:30078
2016-08-03 05:33:41,653 [myid:] - INFO [Thread-0:QuorumPeerConfig@308] - clientPort is not set
2016-08-03 05:33:41,653 [myid:] - INFO [Thread-2:QuorumPeerConfig@332] - secureClientPortAddress is 0.0.0.0/0.0.0.0:30082
2016-08-03 05:33:41,653 [myid:] - INFO [Thread-0:QuorumPeerConfig@332] - secureClientPortAddress is 0.0.0.0/0.0.0.0:30074
2016-08-03 05:33:41,655 [myid:] - INFO [main:ClientBase@248] - server 127.0.0.1:30073 not up
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.client.FourLetterWordMain.send4LetterWord(FourLetterWordMain.java:99)
at org.apache.zookeeper.client.FourLetterWordMain.send4LetterWord(FourLetterWordMain.java:69)
at org.apache.zookeeper.test.ClientBase.waitForServerUp(ClientBase.java:241)
at org.apache.zookeeper.test.ClientBase.waitForServerUp(ClientBase.java:232)
at org.apache.zookeeper.test.SSLTest.testSecureQuorumServer(SSLTest.java:100)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:79)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.TestWatchman$1.evaluate(TestWatchman.java:53)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:38)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:535)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1182)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:1033)
2016-08-03 05:33:41,664 [myid:1] - INFO [Thread-1:DatadirCleanupManager@78] - autopurge.snapRetainCount set to 3
2016-08-03 05:33:41,664 [myid:1] - INFO [Thread-1:DatadirCleanupManager@79] - autopurge.purgeInterval set to 0
2016-08-03 05:33:41,664 [myid:1] - INFO [Thread-1:DatadirCleanupManager@101] - Purge task is not scheduled.
2016-08-03 05:33:41,664 [myid:2] - INFO [Thread-2:DatadirCleanupManager@78] - autopurge.snapRetainCount set to 3
2016-08-03 05:33:41,665 [myid:2] - INFO [Thread-2:DatadirCleanupManager@79] - autopurge.purgeInterval set to 0
2016-08-03 05:33:41,665 [myid:2] - INFO [Thread-2:DatadirCleanupManager@101] - Purge task is not scheduled.
2016-08-03 05:33:41,664 [myid:0] - INFO [Thread-0:DatadirCleanupManager@78] - autopurge.snapRetainCount set to 3
2016-08-03 05:33:41,665 [myid:0] - INFO [Thread-0:DatadirCleanupManager@79] - autopurge.purgeInterval set to 0
2016-08-03 05:33:41,665 [myid:0] - INFO [Thread-0:DatadirCleanupManager@101] - Purge task is not scheduled.
2016-08-03 05:33:41,665 [myid:1] - INFO [Thread-1:ManagedUtil@46] - Log4j found with jmx enabled.
2016-08-03 05:33:41,665 [myid:2] - INFO [Thread-2:ManagedUtil@46] - Log4j found with jmx enabled.
2016-08-03 05:33:41,665 [myid:0] - INFO [Thread-0:ManagedUtil@46] - Log4j found with jmx enabled.
2016-08-03 05:33:41,734 [myid:0] - ERROR [Thread-0:AppenderDynamicMBean@209] - Could not add DynamicLayoutMBean for [CONSOLE,layout=org.apache.log4j.PatternLayout].
javax.management.InstanceAlreadyExistsException: log4j:appender=CONSOLE,layout=org.apache.log4j.PatternLayout
at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:437)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1898)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:966)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324)
at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
at org.apache.log4j.jmx.AbstractDynamicMBean.registerMBean(AbstractDynamicMBean.java:160)
at org.apache.log4j.jmx.AppenderDynamicMBean.registerLayoutMBean(AppenderDynamicMBean.java:203)
at org.apache.log4j.jmx.AppenderDynamicMBean.preRegister(AppenderDynamicMBean.java:339)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.preRegister(DefaultMBeanServerInterceptor.java:1007)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:919)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324)
at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
at org.apache.log4j.jmx.AbstractDynamicMBean.registerMBean(AbstractDynamicMBean.java:160)
at org.apache.log4j.jmx.LoggerDynamicMBean.registerAppenderMBean(LoggerDynamicMBean.java:264)
at org.apache.log4j.jmx.LoggerDynamicMBean.appenderMBeanRegistration(LoggerDynamicMBean.java:252)
at org.apache.log4j.jmx.LoggerDynamicMBean.postRegister(LoggerDynamicMBean.java:280)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.postRegister(DefaultMBeanServerInterceptor.java:1024)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:974)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324)
at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
at org.apache.log4j.jmx.AbstractDynamicMBean.registerMBean(AbstractDynamicMBean.java:160)
at org.apache.log4j.jmx.HierarchyDynamicMBean.addLoggerMBean(HierarchyDynamicMBean.java:125)
at org.apache.log4j.jmx.HierarchyDynamicMBean.postRegister(HierarchyDynamicMBean.java:263)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.postRegister(DefaultMBeanServerInterceptor.java:1024)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:974)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324)
at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
at org.apache.zookeeper.jmx.ManagedUtil.registerLog4jMBeans(ManagedUtil.java:75)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:131)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:120)
at org.apache.zookeeper.server.quorum.QuorumPeerTestBase$MainThread.run(QuorumPeerTestBase.java:245)
at java.lang.Thread.run(Thread.java:745)
2016-08-03 05:33:41,735 [myid:1] - ERROR [Thread-1:HierarchyDynamicMBean@138] - Could not add loggerMBean for [root].
javax.management.InstanceAlreadyExistsException: log4j:logger=root
at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:437)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1898)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:966)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324)
at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
at org.apache.log4j.jmx.AbstractDynamicMBean.registerMBean(AbstractDynamicMBean.java:160)
at org.apache.log4j.jmx.HierarchyDynamicMBean.addLoggerMBean(HierarchyDynamicMBean.java:125)
at org.apache.log4j.jmx.HierarchyDynamicMBean.postRegister(HierarchyDynamicMBean.java:263)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.postRegister(DefaultMBeanServerInterceptor.java:1024)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:974)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324)
at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
at org.apache.zookeeper.jmx.ManagedUtil.registerLog4jMBeans(ManagedUtil.java:75)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:131)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:120)
at org.apache.zookeeper.server.quorum.QuorumPeerTestBase$MainThread.run(QuorumPeerTestBase.java:245)
at java.lang.Thread.run(Thread.java:745)
2016-08-03 05:33:41,734 [myid:2] - ERROR [Thread-2:AppenderDynamicMBean@209] - Could not add DynamicLayoutMBean for [CONSOLE,layout=org.apache.log4j.PatternLayout].
javax.management.InstanceAlreadyExistsException: log4j:appender=CONSOLE,layout=org.apache.log4j.PatternLayout
at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:437)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1898)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:966)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324)
at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
at org.apache.log4j.jmx.AbstractDynamicMBean.registerMBean(AbstractDynamicMBean.java:160)
at org.apache.log4j.jmx.AppenderDynamicMBean.registerLayoutMBean(AppenderDynamicMBean.java:203)
at org.apache.log4j.jmx.AppenderDynamicMBean.preRegister(AppenderDynamicMBean.java:339)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.preRegister(DefaultMBeanServerInterceptor.java:1007)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:919)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324)
at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
at org.apache.log4j.jmx.AbstractDynamicMBean.registerMBean(AbstractDynamicMBean.java:160)
at org.apache.log4j.jmx.LoggerDynamicMBean.registerAppenderMBean(LoggerDynamicMBean.java:264)
at org.apache.log4j.jmx.LoggerDynamicMBean.appenderMBeanRegistration(LoggerDynamicMBean.java:252)
at org.apache.log4j.jmx.LoggerDynamicMBean.postRegister(LoggerDynamicMBean.java:280)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.postRegister(DefaultMBeanServerInterceptor.java:1024)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:974)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324)
at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
at org.apache.log4j.jmx.AbstractDynamicMBean.registerMBean(AbstractDynamicMBean.java:160)
at org.apache.log4j.jmx.HierarchyDynamicMBean.addLoggerMBean(HierarchyDynamicMBean.java:125)
at org.apache.log4j.jmx.HierarchyDynamicMBean.postRegister(HierarchyDynamicMBean.java:263)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.postRegister(DefaultMBeanServerInterceptor.java:1024)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:974)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324)
at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
at org.apache.zookeeper.jmx.ManagedUtil.registerLog4jMBeans(ManagedUtil.java:75)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:131)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:120)
at org.apache.zookeeper.server.quorum.QuorumPeerTestBase$MainThread.run(QuorumPeerTestBase.java:245)
at java.lang.Thread.run(Thread.java:745)
2016-08-03 05:33:41,737 [myid:1] - ERROR [Thread-1:ManagedUtil@114] - Problems while registering log4j jmx beans!
javax.management.InstanceAlreadyExistsException: log4j:hiearchy=default
at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:437)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1898)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:966)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324)
at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
at org.apache.zookeeper.jmx.ManagedUtil.registerLog4jMBeans(ManagedUtil.java:75)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:131)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:120)
at org.apache.zookeeper.server.quorum.QuorumPeerTestBase$MainThread.run(QuorumPeerTestBase.java:245)
at java.lang.Thread.run(Thread.java:745)
2016-08-03 05:33:41,736 [myid:0] - ERROR [Thread-0:LoggerDynamicMBean@270] - Could not add appenderMBean for [CONSOLE].
javax.management.InstanceAlreadyExistsException: log4j:appender=CONSOLE
at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:437)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1898)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:966)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324)
at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
at org.apache.log4j.jmx.AbstractDynamicMBean.registerMBean(AbstractDynamicMBean.java:160)
at org.apache.log4j.jmx.LoggerDynamicMBean.registerAppenderMBean(LoggerDynamicMBean.java:264)
at org.apache.log4j.jmx.LoggerDynamicMBean.appenderMBeanRegistration(LoggerDynamicMBean.java:252)
at org.apache.log4j.jmx.LoggerDynamicMBean.postRegister(LoggerDynamicMBean.java:280)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.postRegister(DefaultMBeanServerInterceptor.java:1024)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:974)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324)
at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
at org.apache.log4j.jmx.AbstractDynamicMBean.registerMBean(AbstractDynamicMBean.java:160)
at org.apache.log4j.jmx.HierarchyDynamicMBean.addLoggerMBean(HierarchyDynamicMBean.java:125)
at org.apache.log4j.jmx.HierarchyDynamicMBean.postRegister(HierarchyDynamicMBean.java:263)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.postRegister(DefaultMBeanServerInterceptor.java:1024)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:974)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324)
at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
at org.apache.zookeeper.jmx.ManagedUtil.registerLog4jMBeans(ManagedUtil.java:75)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:131)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:120)
at org.apache.zookeeper.server.quorum.QuorumPeerTestBase$MainThread.run(QuorumPeerTestBase.java:245)
at java.lang.Thread.run(Thread.java:745)
2016-08-03 05:33:41,739 [myid:1] - WARN [Thread-1:QuorumPeerMain@133] - Unable to register log4j JMX control
javax.management.JMException: javax.management.InstanceAlreadyExistsException: log4j:hiearchy=default
at org.apache.zookeeper.jmx.ManagedUtil.registerLog4jMBeans(ManagedUtil.java:115)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:131)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:120)
at org.apache.zookeeper.server.quorum.QuorumPeerTestBase$MainThread.run(QuorumPeerTestBase.java:245)
at java.lang.Thread.run(Thread.java:745)
2016-08-03 05:33:41,738 [myid:2] - ERROR [Thread-2:LoggerDynamicMBean@270] - Could not add appenderMBean for [CONSOLE].
javax.management.InstanceAlreadyExistsException: log4j:appender=CONSOLE
at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:437)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1898)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:966)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324)
at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
at org.apache.log4j.jmx.AbstractDynamicMBean.registerMBean(AbstractDynamicMBean.java:160)
at org.apache.log4j.jmx.LoggerDynamicMBean.registerAppenderMBean(LoggerDynamicMBean.java:264)
at org.apache.log4j.jmx.LoggerDynamicMBean.appenderMBeanRegistration(LoggerDynamicMBean.java:252)
at org.apache.log4j.jmx.LoggerDynamicMBean.postRegister(LoggerDynamicMBean.java:280)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.postRegister(DefaultMBeanServerInterceptor.java:1024)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:974)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324)
at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
at org.apache.log4j.jmx.AbstractDynamicMBean.registerMBean(AbstractDynamicMBean.java:160)
at org.apache.log4j.jmx.HierarchyDynamicMBean.addLoggerMBean(HierarchyDynamicMBean.java:125)
at org.apache.log4j.jmx.HierarchyDynamicMBean.postRegister(HierarchyDynamicMBean.java:263)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.postRegister(DefaultMBeanServerInterceptor.java:1024)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:974)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324)
at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
at org.apache.zookeeper.jmx.ManagedUtil.registerLog4jMBeans(ManagedUtil.java:75)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:131)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:120)
at org.apache.zookeeper.server.quorum.QuorumPeerTestBase$MainThread.run(QuorumPeerTestBase.java:245)
at java.lang.Thread.run(Thread.java:745)
2016-08-03 05:33:41,740 [myid:1] - INFO [Thread-1:QuorumPeerMain@136] - Starting quorum peer
2016-08-03 05:33:41,742 [myid:2] - ERROR [Thread-2:HierarchyDynamicMBean@138] - Could not add loggerMBean for [root].
javax.management.InstanceAlreadyExistsException: log4j:logger=root
at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:437)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1898)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:966)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324)
at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
at org.apache.log4j.jmx.AbstractDynamicMBean.registerMBean(AbstractDynamicMBean.java:160)
at org.apache.log4j.jmx.HierarchyDynamicMBean.addLoggerMBean(HierarchyDynamicMBean.java:125)
at org.apache.log4j.jmx.HierarchyDynamicMBean.postRegister(HierarchyDynamicMBean.java:263)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.postRegister(DefaultMBeanServerInterceptor.java:1024)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:974)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324)
at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
at org.apache.zookeeper.jmx.ManagedUtil.registerLog4jMBeans(ManagedUtil.java:75)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:131)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:120)
at org.apache.zookeeper.server.quorum.QuorumPeerTestBase$MainThread.run(QuorumPeerTestBase.java:245)
at java.lang.Thread.run(Thread.java:745)
2016-08-03 05:33:41,743 [myid:2] - ERROR [Thread-2:ManagedUtil@114] - Problems while registering log4j jmx beans!
javax.management.InstanceAlreadyExistsException: log4j:hiearchy=default
at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:437)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1898)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:966)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324)
at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
at org.apache.zookeeper.jmx.ManagedUtil.registerLog4jMBeans(ManagedUtil.java:75)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:131)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:120)
at org.apache.zookeeper.server.quorum.QuorumPeerTestBase$MainThread.run(QuorumPeerTestBase.java:245)
at java.lang.Thread.run(Thread.java:745)
2016-08-03 05:33:41,743 [myid:2] - WARN [Thread-2:QuorumPeerMain@133] - Unable to register log4j JMX control
javax.management.JMException: javax.management.InstanceAlreadyExistsException: log4j:hiearchy=default
at org.apache.zookeeper.jmx.ManagedUtil.registerLog4jMBeans(ManagedUtil.java:115)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:131)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:120)
at org.apache.zookeeper.server.quorum.QuorumPeerTestBase$MainThread.run(QuorumPeerTestBase.java:245)
at java.lang.Thread.run(Thread.java:745)
2016-08-03 05:33:41,744 [myid:2] - INFO [Thread-2:QuorumPeerMain@136] - Starting quorum peer
2016-08-03 05:33:41,746 [myid:0] - INFO [Thread-0:QuorumPeerMain@136] - Starting quorum peer
2016-08-03 05:33:41,909 [myid:] - INFO [main:FourLetterWordMain@85] - connecting to 127.0.0.1 30073
2016-08-03 05:33:41,910 [myid:] - INFO [main:ClientBase@248] - server 127.0.0.1:30073 not up
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.client.FourLetterWordMain.send4LetterWord(FourLetterWordMain.java:99)
at org.apache.zookeeper.client.FourLetterWordMain.send4LetterWord(FourLetterWordMain.java:69)
at org.apache.zookeeper.test.ClientBase.waitForServerUp(ClientBase.java:241)
at org.apache.zookeeper.test.ClientBase.waitForServerUp(ClientBase.java:232)
at org.apache.zookeeper.test.SSLTest.testSecureQuorumServer(SSLTest.java:100)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:79)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.TestWatchman$1.evaluate(TestWatchman.java:53)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:38)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:535)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1182)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:1033)
2016-08-03 05:33:41,949 [myid:2] - INFO [Thread-2:QuorumPeer@1327] - Local sessions disabled
2016-08-03 05:33:41,949 [myid:1] - INFO [Thread-1:QuorumPeer@1327] - Local sessions disabled
2016-08-03 05:33:41,949 [myid:1] - INFO [Thread-1:QuorumPeer@1338] - Local session upgrading disabled
2016-08-03 05:33:41,949 [myid:0] - INFO [Thread-0:QuorumPeer@1327] - Local sessions disabled
2016-08-03 05:33:41,950 [myid:0] - INFO [Thread-0:QuorumPeer@1338] - Local session upgrading disabled
2016-08-03 05:33:41,949 [myid:1] - INFO [Thread-1:QuorumPeer@1305] - tickTime set to 4000
2016-08-03 05:33:41,950 [myid:1] - INFO [Thread-1:QuorumPeer@1349] - minSessionTimeout set to 8000
2016-08-03 05:33:41,950 [myid:1] - INFO [Thread-1:QuorumPeer@1360] - maxSessionTimeout set to 80000
2016-08-03 05:33:41,950 [myid:1] - INFO [Thread-1:QuorumPeer@1375] - initLimit set to 10
2016-08-03 05:33:41,949 [myid:2] - INFO [Thread-2:QuorumPeer@1338] - Local session upgrading disabled
2016-08-03 05:33:41,950 [myid:2] - INFO [Thread-2:QuorumPeer@1305] - tickTime set to 4000
2016-08-03 05:33:41,950 [myid:2] - INFO [Thread-2:QuorumPeer@1349] - minSessionTimeout set to 8000
2016-08-03 05:33:41,950 [myid:2] - INFO [Thread-2:QuorumPeer@1360] - maxSessionTimeout set to 80000
2016-08-03 05:33:41,950 [myid:2] - INFO [Thread-2:QuorumPeer@1375] - initLimit set to 10
2016-08-03 05:33:41,950 [myid:0] - INFO [Thread-0:QuorumPeer@1305] - tickTime set to 4000
2016-08-03 05:33:41,951 [myid:0] - INFO [Thread-0:QuorumPeer@1349] - minSessionTimeout set to 8000
2016-08-03 05:33:41,951 [myid:0] - INFO [Thread-0:QuorumPeer@1360] - maxSessionTimeout set to 80000
2016-08-03 05:33:41,951 [myid:0] - INFO [Thread-0:QuorumPeer@1375] - initLimit set to 10
2016-08-03 05:33:41,967 [myid:1] - INFO [Thread-1:QuorumPeer@776] - currentEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
2016-08-03 05:33:41,968 [myid:2] - INFO [Thread-2:QuorumPeer@776] - currentEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
2016-08-03 05:33:41,967 [myid:0] - INFO [Thread-0:QuorumPeer@776] - currentEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
2016-08-03 05:33:42,161 [myid:] - INFO [main:FourLetterWordMain@85] - connecting to 127.0.0.1 30073
2016-08-03 05:33:42,162 [myid:] - INFO [main:ClientBase@248] - server 127.0.0.1:30073 not up
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.client.FourLetterWordMain.send4LetterWord(FourLetterWordMain.java:99)
at org.apache.zookeeper.client.FourLetterWordMain.send4LetterWord(FourLetterWordMain.java:69)
at org.apache.zookeeper.test.ClientBase.waitForServerUp(ClientBase.java:241)
at org.apache.zookeeper.test.ClientBase.waitForServerUp(ClientBase.java:232)
at org.apache.zookeeper.test.SSLTest.testSecureQuorumServer(SSLTest.java:100)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:79)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.TestWatchman$1.evaluate(TestWatchman.java:53)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:38)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:535)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1182)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:1033)
2016-08-03 05:33:42,413 [myid:0] - INFO [Thread-0:QuorumPeer@791] - acceptedEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
2016-08-03 05:33:42,413 [myid:1] - INFO [Thread-1:QuorumPeer@791] - acceptedEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
2016-08-03 05:33:42,413 [myid:] - INFO [main:FourLetterWordMain@85] - connecting to 127.0.0.1 30073
2016-08-03 05:33:42,413 [myid:2] - INFO [Thread-2:QuorumPeer@791] - acceptedEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
2016-08-03 05:33:42,414 [myid:] - INFO [main:ClientBase@248] - server 127.0.0.1:30073 not up
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.client.FourLetterWordMain.send4LetterWord(FourLetterWordMain.java:99)
at org.apache.zookeeper.client.FourLetterWordMain.send4LetterWord(FourLetterWordMain.java:69)
at org.apache.zookeeper.test.ClientBase.waitForServerUp(ClientBase.java:241)
at org.apache.zookeeper.test.ClientBase.waitForServerUp(ClientBase.java:232)
at org.apache.zookeeper.test.SSLTest.testSecureQuorumServer(SSLTest.java:100)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:79)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.TestWatchman$1.evaluate(TestWatchman.java:53)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:38)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:535)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1182)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:1033)
2016-08-03 05:33:42,463 [myid:0] - INFO [Thread-0:NettyServerCnxnFactory@487] - binding to port localhost/127.0.0.1:30073
2016-08-03 05:33:42,463 [myid:2] - INFO [Thread-2:NettyServerCnxnFactory@487] - binding to port localhost/127.0.0.1:30081
2016-08-03 05:33:42,470 [myid:1] - INFO [Thread-1:NettyServerCnxnFactory@487] - binding to port localhost/127.0.0.1:30077
2016-08-03 05:33:42,488 [myid:1] - INFO [Thread-1:NettyServerCnxnFactory@487] - binding to port 0.0.0.0/0.0.0.0:30078
2016-08-03 05:33:42,488 [myid:2] - INFO [Thread-2:NettyServerCnxnFactory@487] - binding to port 0.0.0.0/0.0.0.0:30082
2016-08-03 05:33:42,488 [myid:0] - INFO [Thread-0:NettyServerCnxnFactory@487] - binding to port 0.0.0.0/0.0.0.0:30074
2016-08-03 05:33:42,497 [myid:0] - INFO [QuorumPeerListener:QuorumCnxManager$Listener@632] - My election bind port: localhost/127.0.0.1:30076
2016-08-03 05:33:42,498 [myid:2] - INFO [QuorumPeerListener:QuorumCnxManager$Listener@632] - My election bind port: localhost/127.0.0.1:30084
2016-08-03 05:33:42,499 [myid:1] - INFO [QuorumPeerListener:QuorumCnxManager$Listener@632] - My election bind port: localhost/127.0.0.1:30080
2016-08-03 05:33:42,506 [myid:2] - INFO [QuorumPeer[myid=2](plain=localhost/127.0.0.1:30081)(secure=0.0.0.0/0.0.0.0:30082):QuorumPeer@1033] - LOOKING
2016-08-03 05:33:42,506 [myid:1] - INFO [QuorumPeer[myid=1](plain=localhost/127.0.0.1:30077)(secure=0.0.0.0/0.0.0.0:30078):QuorumPeer@1033] - LOOKING
2016-08-03 05:33:42,507 [myid:0] - INFO [QuorumPeer[myid=0](plain=localhost/127.0.0.1:30073)(secure=0.0.0.0/0.0.0.0:30074):QuorumPeer@1033] - LOOKING
2016-08-03 05:33:42,508 [myid:2] - INFO [QuorumPeer[myid=2](plain=localhost/127.0.0.1:30081)(secure=0.0.0.0/0.0.0.0:30082):FastLeaderElection@894] - New election. My id = 2, proposed zxid=0x0
2016-08-03 05:33:42,508 [myid:1] - INFO [QuorumPeer[myid=1](plain=localhost/127.0.0.1:30077)(secure=0.0.0.0/0.0.0.0:30078):FastLeaderElection@894] - New election. My id = 1, proposed zxid=0x0
2016-08-03 05:33:42,508 [myid:0] - INFO [QuorumPeer[myid=0](plain=localhost/127.0.0.1:30073)(secure=0.0.0.0/0.0.0.0:30074):FastLeaderElection@894] - New election. My id = 0, proposed zxid=0x0
2016-08-03 05:33:42,510 [myid:0] - INFO [localhost/127.0.0.1:30076:QuorumCnxManager$Listener@638] - Received connection request /127.0.0.1:51833
2016-08-03 05:33:42,510 [myid:1] - INFO [localhost/127.0.0.1:30080:QuorumCnxManager$Listener@638] - Received connection request /127.0.0.1:59407
2016-08-03 05:33:42,511 [myid:0] - INFO [WorkerSender[myid=0]:QuorumCnxManager@276] - Have smaller server identifier, so dropping the connection: (1, 0)
2016-08-03 05:33:42,512 [myid:0] - INFO [WorkerSender[myid=0]:QuorumCnxManager@276] - Have smaller server identifier, so dropping the connection: (2, 0)
2016-08-03 05:33:42,512 [myid:2] - INFO [localhost/127.0.0.1:30084:QuorumCnxManager$Listener@638] - Received connection request /127.0.0.1:38675
2016-08-03 05:33:42,512 [myid:0] - INFO [WorkerReceiver[myid=0]:FastLeaderElection@688] - Notification: 2 (message format version), 0 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 0 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version)
2016-08-03 05:33:42,512 [myid:0] - INFO [localhost/127.0.0.1:30076:QuorumCnxManager$Listener@638] - Received connection request /127.0.0.1:51832
2016-08-03 05:33:42,513 [myid:2] - WARN [SendWorker:0:QuorumCnxManager$SendWorker@841] - Send worker leaving thread id 0 my id = 2
2016-08-03 05:33:42,514 [myid:1] - INFO [WorkerSender[myid=1]:QuorumCnxManager@276] - Have smaller server identifier, so dropping the connection: (2, 1)
2016-08-03 05:33:42,513 [myid:2] - WARN [RecvWorker:0:QuorumCnxManager$RecvWorker@915] - Connection broken for id 0, my id = 2, error =
java.net.SocketException: Socket closed
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at java.net.SocketInputStream.read(SocketInputStream.java:210)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:900)
2016-08-03 05:33:42,514 [myid:0] - WARN [SendWorker:2:QuorumCnxManager$SendWorker@837] - Exception when using channel: for id 2 my id = 0 error = java.net.SocketException: Broken pipe
2016-08-03 05:33:42,514 [myid:0] - INFO [WorkerReceiver[myid=0]:FastLeaderElection@688] - Notification: 2 (message format version), 1 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version)
2016-08-03 05:33:42,514 [myid:0] - WARN [RecvWorker:2:QuorumCnxManager$RecvWorker@915] - Connection broken for id 2, my id = 0, error =
java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:392)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:900)
2016-08-03 05:33:42,515 [myid:0] - WARN [RecvWorker:2:QuorumCnxManager$RecvWorker@918] - Interrupting SendWorker
2016-08-03 05:33:42,514 [myid:1] - INFO [localhost/127.0.0.1:30080:QuorumCnxManager$Listener@638] - Received connection request /127.0.0.1:59410
2016-08-03 05:33:42,515 [myid:2] - INFO [localhost/127.0.0.1:30084:QuorumCnxManager$Listener@638] - Received connection request /127.0.0.1:38676
2016-08-03 05:33:42,515 [myid:0] - INFO [localhost/127.0.0.1:30076:QuorumCnxManager$Listener@638] - Received connection request /127.0.0.1:51839
2016-08-03 05:33:42,514 [myid:0] - WARN [SendWorker:2:QuorumCnxManager$SendWorker@841] - Send worker leaving thread id 2 my id = 0
2016-08-03 05:33:42,514 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@688] - Notification: 2 (message format version), 1 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version)
2016-08-03 05:33:42,516 [myid:2] - WARN [SendWorker:1:QuorumCnxManager$SendWorker@832] - Interrupted while waiting for message on queue
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:982)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:63)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:820)
2016-08-03 05:33:42,514 [myid:2] - WARN [RecvWorker:0:QuorumCnxManager$RecvWorker@918] - Interrupting SendWorker
2016-08-03 05:33:42,516 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@688] - Notification: 2 (message format version), 0 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 0 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version)
2016-08-03 05:33:42,516 [myid:2] - WARN [SendWorker:1:QuorumCnxManager$SendWorker@841] - Send worker leaving thread id 1 my id = 2
2016-08-03 05:33:42,516 [myid:1] - INFO [localhost/127.0.0.1:30080:QuorumCnxManager$Listener@638] - Received connection request /127.0.0.1:59413
2016-08-03 05:33:42,516 [myid:0] - INFO [WorkerSender[myid=0]:QuorumCnxManager@276] - Have smaller server identifier, so dropping the connection: (2, 0)
2016-08-03 05:33:42,516 [myid:1] - WARN [SendWorker:2:QuorumCnxManager$SendWorker@837] - Exception when using channel: for id 2 my id = 1 error = java.net.SocketException: Broken pipe
2016-08-03 05:33:42,516 [myid:1] - WARN [RecvWorker:2:QuorumCnxManager$RecvWorker@915] - Connection broken for id 2, my id = 1, error =
java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:392)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:900)
2016-08-03 05:33:42,517 [myid:1] - WARN [RecvWorker:2:QuorumCnxManager$RecvWorker@918] - Interrupting SendWorker
2016-08-03 05:33:42,516 [myid:2] - WARN [RecvWorker:1:QuorumCnxManager$RecvWorker@915] - Connection broken for id 1, my id = 2, error =
java.net.SocketException: Socket closed
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at java.net.SocketInputStream.read(SocketInputStream.java:210)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:900)
2016-08-03 05:33:42,518 [myid:2] - WARN [RecvWorker:1:QuorumCnxManager$RecvWorker@918] - Interrupting SendWorker
2016-08-03 05:33:42,515 [myid:0] - INFO [WorkerReceiver[myid=0]:FastLeaderElection@688] - Notification: 2 (message format version), 1 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 0 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version)
2016-08-03 05:33:42,515 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@688] - Notification: 2 (message format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version)
2016-08-03 05:33:42,517 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@688] - Notification: 2 (message format version), 1 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 0 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version)
2016-08-03 05:33:42,517 [myid:1] - WARN [SendWorker:2:QuorumCnxManager$SendWorker@841] - Send worker leaving thread id 2 my id = 1
2016-08-03 05:33:42,516 [myid:2] - INFO [localhost/127.0.0.1:30084:QuorumCnxManager$Listener@638] - Received connection request /127.0.0.1:38679
2016-08-03 05:33:42,519 [myid:0] - INFO [WorkerReceiver[myid=0]:FastLeaderElection@688] - Notification: 2 (message format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version)
2016-08-03 05:33:42,519 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@688] - Notification: 2 (message format version), 1 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 0 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version)
2016-08-03 05:33:42,519 [myid:2] - WARN [RecvWorker:0:QuorumCnxManager$RecvWorker@915] - Connection broken for id 0, my id = 2, error =
java.net.SocketException: Socket closed
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at java.net.SocketInputStream.read(SocketInputStream.java:210)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:900)
2016-08-03 05:33:42,519 [myid:2] - WARN [RecvWorker:0:QuorumCnxManager$RecvWorker@918] - Interrupting SendWorker
2016-08-03 05:33:42,519 [myid:0] - WARN [RecvWorker:2:QuorumCnxManager$RecvWorker@915] - Connection broken for id 2, my id = 0, error =
java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:392)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:900)
2016-08-03 05:33:42,520 [myid:0] - WARN [RecvWorker:2:QuorumCnxManager$RecvWorker@918] - Interrupting SendWorker
2016-08-03 05:33:42,519 [myid:2] - WARN [SendWorker:0:QuorumCnxManager$SendWorker@832] - Interrupted while waiting for message on queue
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:982)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:63)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:820)
2016-08-03 05:33:42,520 [myid:2] - WARN [SendWorker:0:QuorumCnxManager$SendWorker@841] - Send worker leaving thread id 0 my id = 2
2016-08-03 05:33:42,520 [myid:0] - INFO [WorkerReceiver[myid=0]:FastLeaderElection@688] - Notification: 2 (message format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 0 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version)
2016-08-03 05:33:42,520 [myid:0] - WARN [SendWorker:2:QuorumCnxManager$SendWorker@832] - Interrupted while waiting for message on queue
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:982)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:63)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:820)
2016-08-03 05:33:42,521 [myid:0] - WARN [SendWorker:2:QuorumCnxManager$SendWorker@841] - Send worker leaving thread id 2 my id = 0
2016-08-03 05:33:42,520 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@688] - Notification: 2 (message format version), 1 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version)
2016-08-03 05:33:42,519 [myid:0] - INFO [localhost/127.0.0.1:30076:QuorumCnxManager$Listener@638] - Received connection request /127.0.0.1:51842
2016-08-03 05:33:42,519 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@688] - Notification: 2 (message format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version)
2016-08-03 05:33:42,521 [myid:0] - INFO [WorkerSender[myid=0]:QuorumCnxManager@276] - Have smaller server identifier, so dropping the connection: (2, 0)
2016-08-03 05:33:42,521 [myid:2] - INFO [localhost/127.0.0.1:30084:QuorumCnxManager$Listener@638] - Received connection request /127.0.0.1:38682
2016-08-03 05:33:42,522 [myid:0] - WARN [RecvWorker:2:QuorumCnxManager$RecvWorker@915] - Connection broken for id 2, my id = 0, error =
java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:392)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:900)
2016-08-03 05:33:42,522 [myid:0] - INFO [localhost/127.0.0.1:30076:QuorumCnxManager$Listener@638] - Received connection request /127.0.0.1:51844
2016-08-03 05:33:42,522 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@688] - Notification: 2 (message format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version)
2016-08-03 05:33:42,522 [myid:2] - WARN [RecvWorker:0:QuorumCnxManager$RecvWorker@915] - Connection broken for id 0, my id = 2, error =
java.net.SocketException: Socket closed
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at java.net.SocketInputStream.read(SocketInputStream.java:210)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:900)
2016-08-03 05:33:42,523 [myid:2] - WARN [RecvWorker:0:QuorumCnxManager$RecvWorker@918] - Interrupting SendWorker
2016-08-03 05:33:42,522 [myid:2] - WARN [SendWorker:0:QuorumCnxManager$SendWorker@832] - Interrupted while waiting for message on queue
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:982)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:63)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:820)
2016-08-03 05:33:42,524 [myid:2] - WARN [SendWorker:0:QuorumCnxManager$SendWorker@841] - Send worker leaving thread id 0 my id = 2
2016-08-03 05:33:42,524 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@688] - Notification: 2 (message format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 0 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version)
2016-08-03 05:33:42,523 [myid:0] - WARN [SendWorker:2:QuorumCnxManager$SendWorker@832] - Interrupted while waiting for message on queue
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:982)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:63)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:820)
2016-08-03 05:33:42,523 [myid:0] - INFO [WorkerReceiver[myid=0]:FastLeaderElection@688] - Notification: 2 (message format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version)
2016-08-03 05:33:42,523 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@688] - Notification: 2 (message format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 0 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version)
2016-08-03 05:33:42,522 [myid:0] - WARN [RecvWorker:2:QuorumCnxManager$RecvWorker@918] - Interrupting SendWorker
2016-08-03 05:33:42,524 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@688] - Notification: 2 (message format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version)
2016-08-03 05:33:42,524 [myid:0] - WARN [SendWorker:2:QuorumCnxManager$SendWorker@841] - Send worker leaving thread id 2 my id = 0
2016-08-03 05:33:42,525 [myid:0] - INFO [WorkerReceiver[myid=0]:FastLeaderElection@688] - Notification: 2 (message format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version)
2016-08-03 05:33:42,525 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@688] - Notification: 2 (message format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version)
2016-08-03 05:33:42,526 [myid:0] - INFO [WorkerReceiver[myid=0]:FastLeaderElection@688] - Notification: 2 (message format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version)
2016-08-03 05:33:42,526 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@688] - Notification: 2 (message format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 0 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version)
2016-08-03 05:33:42,665 [myid:] - INFO [main:FourLetterWordMain@85] - connecting to 127.0.0.1 30073
2016-08-03 05:33:42,701 [myid:0] - INFO [New I/O worker #1:NettyServerCnxn@275] - Processing stat command from /127.0.0.1:60036
2016-08-03 05:33:42,725 [myid:1] - INFO [QuorumPeer[myid=1](plain=localhost/127.0.0.1:30077)(secure=0.0.0.0/0.0.0.0:30078):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id1,name1=replica.1,name2=LeaderElection]
2016-08-03 05:33:42,726 [myid:1] - INFO [QuorumPeer[myid=1](plain=localhost/127.0.0.1:30077)(secure=0.0.0.0/0.0.0.0:30078):QuorumPeer@1109] - FOLLOWING
2016-08-03 05:33:42,726 [myid:0] - INFO [QuorumPeer[myid=0](plain=localhost/127.0.0.1:30073)(secure=0.0.0.0/0.0.0.0:30074):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id0,name1=replica.0,name2=LeaderElection]
2016-08-03 05:33:42,726 [myid:2] - INFO [QuorumPeer[myid=2](plain=localhost/127.0.0.1:30081)(secure=0.0.0.0/0.0.0.0:30082):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id2,name1=replica.2,name2=LeaderElection]
2016-08-03 05:33:42,727 [myid:0] - INFO [QuorumPeer[myid=0](plain=localhost/127.0.0.1:30073)(secure=0.0.0.0/0.0.0.0:30074):QuorumPeer@1109] - FOLLOWING
2016-08-03 05:33:42,727 [myid:2] - INFO [QuorumPeer[myid=2](plain=localhost/127.0.0.1:30081)(secure=0.0.0.0/0.0.0.0:30082):QuorumPeer@1121] - LEADING
2016-08-03 05:33:42,730 [myid:2] - INFO [QuorumPeer[myid=2](plain=localhost/127.0.0.1:30081)(secure=0.0.0.0/0.0.0.0:30082):Leader@63] - TCP NoDelay set to: true
2016-08-03 05:33:42,730 [myid:2] - INFO [QuorumPeer[myid=2](plain=localhost/127.0.0.1:30081)(secure=0.0.0.0/0.0.0.0:30082):Leader@83] - zookeeper.leader.maxConcurrentSnapshots = 10
2016-08-03 05:33:42,730 [myid:2] - INFO [QuorumPeer[myid=2](plain=localhost/127.0.0.1:30081)(secure=0.0.0.0/0.0.0.0:30082):Leader@85] - zookeeper.leader.maxConcurrentSnapshotTimeout = 5
2016-08-03 05:33:42,731 [myid:1] - INFO [QuorumPeer[myid=1](plain=localhost/127.0.0.1:30077)(secure=0.0.0.0/0.0.0.0:30078):Learner@88] - TCP NoDelay set to: true
2016-08-03 05:33:42,737 [myid:0] - INFO [QuorumPeer[myid=0](plain=localhost/127.0.0.1:30073)(secure=0.0.0.0/0.0.0.0:30074):Environment@109] - Server environment:zookeeper.version=3.6.0-SNAPSHOT-1755017, built on 08/03/2016 05:24 GMT
2016-08-03 05:33:42,738 [myid:0] - INFO [QuorumPeer[myid=0](plain=localhost/127.0.0.1:30073)(secure=0.0.0.0/0.0.0.0:30074):Environment@109] - Server environment:host.name=asf909.gq1.ygridcore.net
2016-08-03 05:33:42,738 [myid:0] - INFO [QuorumPeer[myid=0](plain=localhost/127.0.0.1:30073)(secure=0.0.0.0/0.0.0.0:30074):Environment@109] - Server environment:java.version=1.7.0_80
2016-08-03 05:33:42,738 [myid:0] - INFO [QuorumPeer[myid=0](plain=localhost/127.0.0.1:30073)(secure=0.0.0.0/0.0.0.0:30074):Environment@109] - Server environment:java.vendor=Oracle Corporation
2016-08-03 05:33:42,738 [myid:0] - INFO [QuorumPeer[myid=0](plain=localhost/127.0.0.1:30073)(secure=0.0.0.0/0.0.0.0:30074):Environment@109] - Server environment:java.home=/home/jenkins/jenkins-slave/tools/hudson.model.JDK/latest1.7/jre
2016-08-03 05:33:42,738 [myid:0] - INFO [QuorumPeer[myid=0](plain=localhost/127.0.0.1:30073)(secure=0.0.0.0/0.0.0.0:30074):Environment@109] - Server environment:java.class.path=/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/trunk/build/test/classes:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/trunk/build/test/lib/antlr-2.7.7.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/trunk/build/test/lib/antlr4-runtime-4.5.1-1.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/trunk/build/test/lib/checkstyle-6.13.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/trunk/build/test/lib/commons-beanutils-1.9.2.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/trunk/build/test/lib/commons-cli-1.3.1.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/trunk/build/test/lib/commons-collections-3.2.2.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/trunk/build/test/lib/commons-lang3-3.4.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/trunk/build/test/lib/commons-logging-1.1.1.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/trunk/build/test/lib/guava-18.0.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/trunk/build/test/lib/hamcrest-core-1.3.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/trunk/build/test/lib/junit-4.12.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/trunk/build/test/lib/mockito-all-1.8.2.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/trunk/build/classes:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/trunk/src/java/lib/ivy-2.4.0.jar:/home/jenkins/tools/ant/latest/lib/ant.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/trunk/build/lib/apache-rat-core-0.10.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/trunk/build/lib/apache-rat-tasks-0.10.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/trunk/build/lib/commons-cli-1.2.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/trunk/build/lib/commons-collections-3.2.2.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/trunk/build/lib/commons-compress-1.5.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/trunk/build/lib/commons-io-2.2.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/trunk/build/lib/commons-lang-2.6.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/trunk/build/lib/jackson-core-asl-1.9.11.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/trunk/build/lib/jackson-mapper-asl-1.9.11.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/trunk/build/lib/javacc.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/trunk/build/lib/jetty-6.1.26.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/trunk/build/lib/jetty-util-6.1.26.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/trunk/build/lib/jline-2.11.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/trunk/build/lib/log4j-1.2.17.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/trunk/build/lib/netty-3.10.5.Final.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/trunk/build/lib/servlet-api-2.5-20081211.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/trunk/build/lib/slf4j-api-1.7.5.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/trunk/build/lib/slf4j-log4j12-1.7.5.jar:/usr/local/asfpackages/ant/apache-ant-1.9.7/lib/ant-launcher.jar:/home/jenkins/tools/ant/latest/lib/ant-junit.jar:/home/jenkins/tools/ant/latest/lib/ant-junit4.jar
2016-08-03 05:33:42,738 [myid:0] - INFO [QuorumPeer[myid=0](plain=localhost/127.0.0.1:30073)(secure=0.0.0.0/0.0.0.0:30074):Environment@109] - Server environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2016-08-03 05:33:42,738 [myid:0] - INFO [QuorumPeer[myid=0](plain=localhost/127.0.0.1:30073)(secure=0.0.0.0/0.0.0.0:30074):Environment@109] - Server environment:java.io.tmpdir=/tmp
2016-08-03 05:33:42,738 [myid:0] - INFO [QuorumPeer[myid=0](plain=localhost/127.0.0.1:30073)(secure=0.0.0.0/0.0.0.0:30074):Environment@109] - Server environment:java.compiler=<NA>
2016-08-03 05:33:42,738 [myid:0] - INFO [QuorumPeer[myid=0](plain=localhost/127.0.0.1:30073)(secure=0.0.0.0/0.0.0.0:30074):Environment@109] - Server environment:os.name=Linux
2016-08-03 05:33:42,738 [myid:0] - INFO [QuorumPeer[myid=0](plain=localhost/127.0.0.1:30073)(secure=0.0.0.0/0.0.0.0:30074):Environment@109] - Server environment:os.arch=amd64
2016-08-03 05:33:42,739 [myid:0] - INFO [QuorumPeer[myid=0](plain=localhost/127.0.0.1:30073)(secure=0.0.0.0/0.0.0.0:30074):Environment@109] - Server environment:os.version=3.13.0-36-lowlatency
2016-08-03 05:33:42,739 [myid:0] - INFO [QuorumPeer[myid=0](plain=localhost/127.0.0.1:30073)(secure=0.0.0.0/0.0.0.0:30074):Environment@109] - Server environment:user.name=jenkins
2016-08-03 05:33:42,739 [myid:0] - INFO [QuorumPeer[myid=0](plain=localhost/127.0.0.1:30073)(secure=0.0.0.0/0.0.0.0:30074):Environment@109] - Server environment:user.home=/home/jenkins
2016-08-03 05:33:42,739 [myid:0] - INFO [QuorumPeer[myid=0](plain=localhost/127.0.0.1:30073)(secure=0.0.0.0/0.0.0.0:30074):Environment@109] - Server environment:user.dir=/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/trunk/build/test
2016-08-03 05:33:42,739 [myid:0] - INFO [QuorumPeer[myid=0](plain=localhost/127.0.0.1:30073)(secure=0.0.0.0/0.0.0.0:30074):Environment@109] - Server environment:os.memory.free=437MB
2016-08-03 05:33:42,739 [myid:0] - INFO [QuorumPeer[myid=0](plain=localhost/127.0.0.1:30073)(secure=0.0.0.0/0.0.0.0:30074):Environment@109] - Server environment:os.memory.max=491MB
2016-08-03 05:33:42,739 [myid:0] - INFO [QuorumPeer[myid=0](plain=localhost/127.0.0.1:30073)(secure=0.0.0.0/0.0.0.0:30074):Environment@109] - Server environment:os.memory.total=491MB
2016-08-03 05:33:42,741 [myid:0] - INFO [QuorumPeer[myid=0](plain=localhost/127.0.0.1:30073)(secure=0.0.0.0/0.0.0.0:30074):ZooKeeperServer@858] - minSessionTimeout set to 8000
2016-08-03 05:33:42,741 [myid:2] - INFO [QuorumPeer[myid=2](plain=localhost/127.0.0.1:30081)(secure=0.0.0.0/0.0.0.0:30082):ZooKeeperServer@858] - minSessionTimeout set to 8000
2016-08-03 05:33:42,741 [myid:2] - INFO [QuorumPeer[myid=2](plain=localhost/127.0.0.1:30081)(secure=0.0.0.0/0.0.0.0:30082):ZooKeeperServer@867] - maxSessionTimeout set to 80000
2016-08-03 05:33:42,741 [myid:1] - INFO [QuorumPeer[myid=1](plain=localhost/127.0.0.1:30077)(secure=0.0.0.0/0.0.0.0:30078):ZooKeeperServer@858] - minSessionTimeout set to 8000
2016-08-03 05:33:42,741 [myid:1] - INFO [QuorumPeer[myid=1](plain=localhost/127.0.0.1:30077)(secure=0.0.0.0/0.0.0.0:30078):ZooKeeperServer@867] - maxSessionTimeout set to 80000
2016-08-03 05:33:42,741 [myid:2] - INFO [QuorumPeer[myid=2](plain=localhost/127.0.0.1:30081)(secure=0.0.0.0/0.0.0.0:30082):ZooKeeperServer@156] - Created server with tickTime 4000 minSessionTimeout 8000 maxSessionTimeout 80000 datadir /home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/trunk/build/test/tmp/test4754133370740902988.junit.dir/data/version-2 snapdir /home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/trunk/build/test/tmp/test4754133370740902988.junit.dir/data/version-2
2016-08-03 05:33:42,741 [myid:0] - INFO [QuorumPeer[myid=0](plain=localhost/127.0.0.1:30073)(secure=0.0.0.0/0.0.0.0:30074):ZooKeeperServer@867] - maxSessionTimeout set to 80000
2016-08-03 05:33:42,742 [myid:0] - INFO [QuorumPeer[myid=0](plain=localhost/127.0.0.1:30073)(secure=0.0.0.0/0.0.0.0:30074):ZooKeeperServer@156] - Created server with tickTime 4000 minSessionTimeout 8000 maxSessionTimeout 80000 datadir /home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/trunk/build/test/tmp/test8166855371662584608.junit.dir/data/version-2 snapdir /home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/trunk/build/test/tmp/test8166855371662584608.junit.dir/data/version-2
2016-08-03 05:33:42,741 [myid:1] - INFO [QuorumPeer[myid=1](plain=localhost/127.0.0.1:30077)(secure=0.0.0.0/0.0.0.0:30078):ZooKeeperServer@156] - Created server with tickTime 4000 minSessionTimeout 8000 maxSessionTimeout 80000 datadir /home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/trunk/build/test/tmp/test5311610672905591522.junit.dir/data/version-2 snapdir /home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/trunk/build/test/tmp/test5311610672905591522.junit.dir/data/version-2
2016-08-03 05:33:42,742 [myid:0] - INFO [QuorumPeer[myid=0](plain=localhost/127.0.0.1:30073)(secure=0.0.0.0/0.0.0.0:30074):Follower@66] - FOLLOWING - LEADER ELECTION TOOK - 15 MS
2016-08-03 05:33:42,742 [myid:1] - INFO [QuorumPeer[myid=1](plain=localhost/127.0.0.1:30077)(secure=0.0.0.0/0.0.0.0:30078):Follower@66] - FOLLOWING - LEADER ELECTION TOOK - 16 MS
2016-08-03 05:33:42,743 [myid:2] - INFO [QuorumPeer[myid=2](plain=localhost/127.0.0.1:30081)(secure=0.0.0.0/0.0.0.0:30082):Leader@412] - LEADING - LEADER ELECTION TOOK - 16 MS
2016-08-03 05:33:42,747 [myid:2] - INFO [QuorumPeer[myid=2](plain=localhost/127.0.0.1:30081)(secure=0.0.0.0/0.0.0.0:30082):FileTxnSnapLog@298] - Snapshotting: 0x0 to /home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/trunk/build/test/tmp/test4754133370740902988.junit.dir/data/version-2/snapshot.0
2016-08-03 05:33:42,764 [myid:2] - INFO [LearnerHandler-/127.0.0.1:38806:LearnerHandler@382] - Follower sid: 1 : info : localhost:30079:30080:participant;localhost:30077
2016-08-03 05:33:42,764 [myid:2] - INFO [LearnerHandler-/127.0.0.1:38807:LearnerHandler@382] - Follower sid: 0 : info : localhost:30075:30076:participant;localhost:30073
2016-08-03 05:33:42,961 [myid:] - INFO [main:FourLetterWordMain@85] - connecting to 127.0.0.1 30073
2016-08-03 05:33:42,962 [myid:0] - INFO [New I/O worker #5:NettyServerCnxn@275] - Processing stat command from /127.0.0.1:60040
2016-08-03 05:33:43,213 [myid:] - INFO [main:FourLetterWordMain@85] - connecting to 127.0.0.1 30073
2016-08-03 05:33:43,215 [myid:0] - INFO [New I/O worker #8:NettyServerCnxn@275] - Processing stat command from /127.0.0.1:60043
2016-08-03 05:33:43,465 [myid:] - INFO [main:FourLetterWordMain@85] - connecting to 127.0.0.1 30073
2016-08-03 05:33:43,467 [myid:0] - INFO [New I/O worker #12:NettyServerCnxn@275] - Processing stat command from /127.0.0.1:60044
2016-08-03 05:33:43,718 [myid:] - INFO [main:FourLetterWordMain@85] - connecting to 127.0.0.1 30073
2016-08-03 05:33:43,719 [myid:0] - INFO [New I/O worker #15:NettyServerCnxn@275] - Processing stat command from /127.0.0.1:60047
2016-08-03 05:33:43,725 [myid:2] - INFO [LearnerHandler-/127.0.0.1:38806:LearnerHandler@683] - Synchronizing with Follower sid: 1 maxCommittedLog=0x0 minCommittedLog=0x0 lastProcessedZxid=0x0 peerLastZxid=0x0
2016-08-03 05:33:43,725 [myid:2] - INFO [LearnerHandler-/127.0.0.1:38806:LearnerHandler@727] - Sending DIFF zxid=0x0 for peer sid: 1
2016-08-03 05:33:43,727 [myid:1] - INFO [QuorumPeer[myid=1](plain=localhost/127.0.0.1:30077)(secure=0.0.0.0/0.0.0.0:30078):Learner@366] - Getting a diff from the leader 0x0
2016-08-03 05:33:43,732 [myid:1] - INFO [QuorumPeer[myid=1](plain=localhost/127.0.0.1:30077)(secure=0.0.0.0/0.0.0.0:30078):Learner@509] - Learner received NEWLEADER message
2016-08-03 05:33:43,783 [myid:2] - INFO [LearnerHandler-/127.0.0.1:38807:LearnerHandler@683] - Synchronizing with Follower sid: 0 maxCommittedLog=0x0 minCommittedLog=0x0 lastProcessedZxid=0x0 peerLastZxid=0x0
2016-08-03 05:33:43,783 [myid:2] - INFO [LearnerHandler-/127.0.0.1:38807:LearnerHandler@727] - Sending DIFF zxid=0x0 for peer sid: 0
2016-08-03 05:33:43,784 [myid:0] - INFO [QuorumPeer[myid=0](plain=localhost/127.0.0.1:30073)(secure=0.0.0.0/0.0.0.0:30074):Learner@366] - Getting a diff from the leader 0x0
2016-08-03 05:33:43,784 [myid:0] - INFO [QuorumPeer[myid=0](plain=localhost/127.0.0.1:30073)(secure=0.0.0.0/0.0.0.0:30074):Learner@509] - Learner received NEWLEADER message
2016-08-03 05:33:43,908 [myid:1] - INFO [QuorumPeer[myid=1](plain=localhost/127.0.0.1:30077)(secure=0.0.0.0/0.0.0.0:30078):FileTxnSnapLog@298] - Snapshotting: 0x0 to /home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/trunk/build/test/tmp/test5311610672905591522.junit.dir/data/version-2/snapshot.0
2016-08-03 05:33:43,908 [myid:0] - INFO [QuorumPeer[myid=0](plain=localhost/127.0.0.1:30073)(secure=0.0.0.0/0.0.0.0:30074):FileTxnSnapLog@298] - Snapshotting: 0x0 to /home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/trunk/build/test/tmp/test8166855371662584608.junit.dir/data/version-2/snapshot.0
2016-08-03 05:33:43,964 [myid:2] - INFO [QuorumPeer[myid=2](plain=localhost/127.0.0.1:30081)(secure=0.0.0.0/0.0.0.0:30082):Leader@1249] - Have quorum of supporters, sids: [ [0, 2],[0, 2] ]; starting up and setting last processed zxid: 0x100000000
2016-08-03 05:33:43,970 [myid:] - INFO [main:FourLetterWordMain@85] - connecting to 127.0.0.1 30073
2016-08-03 05:33:43,971 [myid:0] - INFO [New I/O worker #17:NettyServerCnxn@275] - Processing stat command from /127.0.0.1:60050
2016-08-03 05:33:44,222 [myid:] - INFO [main:FourLetterWordMain@85] - connecting to 127.0.0.1 30073
2016-08-03 05:33:44,223 [myid:0] - INFO [New I/O worker #20:NettyServerCnxn@275] - Processing stat command from /127.0.0.1:60051
2016-08-03 05:33:44,474 [myid:] - INFO [main:FourLetterWordMain@85] - connecting to 127.0.0.1 30073
2016-08-03 05:33:44,475 [myid:0] - INFO [New I/O worker #23:NettyServerCnxn@275] - Processing stat command from /127.0.0.1:60055
2016-08-03 05:33:44,726 [myid:] - INFO [main:FourLetterWordMain@85] - connecting to 127.0.0.1 30073
2016-08-03 05:33:44,728 [myid:0] - INFO [New I/O worker #26:NettyServerCnxn@275] - Processing stat command from /127.0.0.1:60056
2016-08-03 05:33:44,729 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@98] - TEST METHOD FAILED testSecureQuorumServer
java.lang.AssertionError: waiting for server 0 being up
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.apache.zookeeper.test.SSLTest.testSecureQuorumServer(SSLTest.java:100)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:79)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.TestWatchman$1.evaluate(TestWatchman.java:53)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:38)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:535)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1182)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:1033)
2016-08-03 05:33:44,731 [myid:] - INFO [main:ZKTestCase$1@70] - FAILED testSecureQuorumServer
java.lang.AssertionError: waiting for server 0 being up
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.apache.zookeeper.test.SSLTest.testSecureQuorumServer(SSLTest.java:100)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:79)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.TestWatchman$1.evaluate(TestWatchman.java:53)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:38)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:535)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1182)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:1033)
2016-08-03 05:33:44,732 [myid:] - INFO [main:ZKTestCase$1@60] - FINISHED testSecureQuorumServer
2016-08-03 05:33:44,736 [myid:] - INFO [main:ZKTestCase$1@55] - STARTING testSecureStandaloneServer
2016-08-03 05:33:44,737 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@77] - RUNNING TEST METHOD testSecureStandaloneServer
2016-08-03 05:33:44,737 [myid:] - INFO [main:PortAssignment@85] - Assigned port 30085 from range 30072 - 32764.
2016-08-03 05:33:44,737 [myid:] - INFO [main:QuorumPeerTestBase$MainThread@131] - id = -1 tmpDir = /home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/trunk/build/test/tmp/test1220380102294608498.junit.dir clientPort = -1 adminServerPort = 8080
2016-08-03 05:33:44,738 [myid:] - INFO [Thread-6:QuorumPeerConfig@116] - Reading configuration from: /home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/trunk/build/test/tmp/test1220380102294608498.junit.dir/zoo.cfg
2016-08-03 05:33:44,738 [myid:] - INFO [Thread-6:QuorumPeerConfig@308] - clientPort is not set
2016-08-03 05:33:44,739 [myid:] - INFO [Thread-6:QuorumPeerConfig@332] - secureClientPortAddress is 0.0.0.0/0.0.0.0:30085
2016-08-03 05:33:44,739 [myid:-1] - INFO [Thread-6:DatadirCleanupManager@78] - autopurge.snapRetainCount set to 3
2016-08-03 05:33:44,739 [myid:-1] - INFO [Thread-6:DatadirCleanupManager@79] - autopurge.purgeInterval set to 0
2016-08-03 05:33:44,739 [myid:-1] - INFO [Thread-6:DatadirCleanupManager@101] - Purge task is not scheduled.
2016-08-03 05:33:44,740 [myid:-1] - WARN [Thread-6:QuorumPeerMain@122] - Either no config or no quorum defined in config, running in standalone mode
2016-08-03 05:33:44,741 [myid:-1] - INFO [Thread-6:ManagedUtil@46] - Log4j found with jmx enabled.
2016-08-03 05:33:44,741 [myid:-1] - ERROR [Thread-6:ManagedUtil@114] - Problems while registering log4j jmx beans!
javax.management.InstanceAlreadyExistsException: log4j:hiearchy=default
at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:437)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1898)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:966)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324)
at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
at org.apache.zookeeper.jmx.ManagedUtil.registerLog4jMBeans(ManagedUtil.java:75)
at org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:91)
at org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:61)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:125)
at org.apache.zookeeper.server.quorum.QuorumPeerTestBase$MainThread.run(QuorumPeerTestBase.java:245)
at java.lang.Thread.run(Thread.java:745)
2016-08-03 05:33:44,742 [myid:-1] - WARN [Thread-6:ZooKeeperServerMain@93] - Unable to register log4j JMX control
javax.management.JMException: javax.management.InstanceAlreadyExistsException: log4j:hiearchy=default
at org.apache.zookeeper.jmx.ManagedUtil.registerLog4jMBeans(ManagedUtil.java:115)
at org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:91)
at org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:61)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:125)
at org.apache.zookeeper.server.quorum.QuorumPeerTestBase$MainThread.run(QuorumPeerTestBase.java:245)
at java.lang.Thread.run(Thread.java:745)
2016-08-03 05:33:44,743 [myid:-1] - INFO [Thread-6:QuorumPeerConfig@116] - Reading configuration from: /home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/trunk/build/test/tmp/test1220380102294608498.junit.dir/zoo.cfg
2016-08-03 05:33:44,743 [myid:] - INFO [main:Environment@109] - Client environment:zookeeper.version=3.6.0-SNAPSHOT-1755017, built on 08/03/2016 05:24 GMT
2016-08-03 05:33:44,743 [myid:] - INFO [main:Environment@109] - Client environment:host.name=asf909.gq1.ygridcore.net
2016-08-03 05:33:44,744 [myid:] - INFO [main:Environment@109] - Client environment:java.version=1.7.0_80
2016-08-03 05:33:44,744 [myid:] - INFO [main:Environment@109] - Client environment:java.vendor=Oracle Corporation
2016-08-03 05:33:44,744 [myid:] - INFO [main:Environment@109] - Client environment:java.home=/home/jenkins/jenkins-slave/tools/hudson.model.JDK/latest1.7/jre
2016-08-03 05:33:44,743 [myid:-1] - INFO [Thread-6:QuorumPeerConfig@308] - clientPort is not set
2016-08-03 05:33:44,744 [myid:] - INFO [main:Environment@109] - Client environment:java.class.path=/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/trunk/build/test/classes:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/trunk/build/test/lib/antlr-2.7.7.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/trunk/build/test/lib/antlr4-runtime-4.5.1-1.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/trunk/build/test/lib/checkstyle-6.13.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/trunk/build/test/lib/commons-beanutils-1.9.2.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/trunk/build/test/lib/commons-cli-1.3.1.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/trunk/build/test/lib/commons-collections-3.2.2.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/trunk/build/test/lib/commons-lang3-3.4.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/trunk/build/test/lib/commons-logging-1.1.1.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/trunk/build/test/lib/guava-18.0.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/trunk/build/test/lib/hamcrest-core-1.3.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/trunk/build/test/lib/junit-4.12.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/trunk/build/test/lib/mockito-all-1.8.2.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/trunk/build/classes:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/trunk/src/java/lib/ivy-2.4.0.jar:/home/jenkins/tools/ant/latest/lib/ant.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/trunk/build/lib/apache-rat-core-0.10.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/trunk/build/lib/apache-rat-tasks-0.10.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/trunk/build/lib/commons-cli-1.2.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/trunk/build/lib/commons-collections-3.2.2.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/trunk/build/lib/commons-compress-1.5.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/trunk/build/lib/commons-io-2.2.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/trunk/build/lib/commons-lang-2.6.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/trunk/build/lib/jackson-core-asl-1.9.11.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/trunk/build/lib/jackson-mapper-asl-1.9.11.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/trunk/build/lib/javacc.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/trunk/build/lib/jetty-6.1.26.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/trunk/build/lib/jetty-util-6.1.26.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/trunk/build/lib/jline-2.11.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/trunk/build/lib/log4j-1.2.17.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/trunk/build/lib/netty-3.10.5.Final.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/trunk/build/lib/servlet-api-2.5-20081211.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/trunk/build/lib/slf4j-api-1.7.5.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/trunk/build/lib/slf4j-log4j12-1.7.5.jar:/usr/local/asfpackages/ant/apache-ant-1.9.7/lib/ant-launcher.jar:/home/jenkins/tools/ant/latest/lib/ant-junit.jar:/home/jenkins/tools/ant/latest/lib/ant-junit4.jar
2016-08-03 05:33:44,745 [myid:] - INFO [main:Environment@109] - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2016-08-03 05:33:44,745 [myid:] - INFO [main:Environment@109] - Client environment:java.io.tmpdir=/tmp
2016-08-03 05:33:44,745 [myid:] - INFO [main:Environment@109] - Client environment:java.compiler=<NA>
2016-08-03 05:33:44,745 [myid:] - INFO [main:Environment@109] - Client environment:os.name=Linux
2016-08-03 05:33:44,745 [myid:] - INFO [main:Environment@109] - Client environment:os.arch=amd64
2016-08-03 05:33:44,744 [myid:-1] - INFO [Thread-6:QuorumPeerConfig@332] - secureClientPortAddress is 0.0.0.0/0.0.0.0:30085
2016-08-03 05:33:44,745 [myid:] - INFO [main:Environment@109] - Client environment:os.version=3.13.0-36-lowlatency
2016-08-03 05:33:44,746 [myid:] - INFO [main:Environment@109] - Client environment:user.name=jenkins
2016-08-03 05:33:44,746 [myid:] - INFO [main:Environment@109] - Client environment:user.home=/home/jenkins
2016-08-03 05:33:44,746 [myid:] - INFO [main:Environment@109] - Client environment:user.dir=/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/trunk/build/test
2016-08-03 05:33:44,746 [myid:] - INFO [main:Environment@109] - Client environment:os.memory.free=424MB
2016-08-03 05:33:44,747 [myid:] - INFO [main:Environment@109] - Client environment:os.memory.max=491MB
2016-08-03 05:33:44,747 [myid:] - INFO [main:Environment@109] - Client environment:os.memory.total=491MB
2016-08-03 05:33:44,746 [myid:-1] - INFO [Thread-6:ZooKeeperServerMain@113] - Starting server
2016-08-03 05:33:44,747 [myid:-1] - INFO [Thread-6:ZooKeeperServer@858] - minSessionTimeout set to 8000
2016-08-03 05:33:44,748 [myid:-1] - INFO [Thread-6:ZooKeeperServer@867] - maxSessionTimeout set to 80000
2016-08-03 05:33:44,748 [myid:-1] - INFO [Thread-6:ZooKeeperServer@156] - Created server with tickTime 4000 minSessionTimeout 8000 maxSessionTimeout 80000 datadir /home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/trunk/build/test/tmp/test1220380102294608498.junit.dir/data/version-2 snapdir /home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/trunk/build/test/tmp/test1220380102294608498.junit.dir/data/version-2
2016-08-03 05:33:44,749 [myid:] - INFO [main:ZooKeeper@855] - Initiating client connection, connectString=127.0.0.1:30085 sessionTimeout=3000 watcher=org.apache.zookeeper.test.SSLTest$2@10614f3d
2016-08-03 05:33:44,760 [myid:-1] - INFO [Thread-6:NettyServerCnxnFactory@487] - binding to port 0.0.0.0/0.0.0.0:30085
2016-08-03 05:33:44,761 [myid:-1] - INFO [Thread-6:FileTxnSnapLog@298] - Snapshotting: 0x0 to /home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/trunk/build/test/tmp/test1220380102294608498.junit.dir/data/version-2/snapshot.0
2016-08-03 05:33:44,778 [myid:-1] - INFO [Thread-6:ContainerManager@64] - Using checkIntervalMs=60000 maxPerMinute=10000
2016-08-03 05:33:44,783 [myid:127.0.0.1:30085] - INFO [main-SendThread(127.0.0.1:30085):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:30085. Will not attempt to authenticate using SASL (unknown error)
2016-08-03 05:33:45,065 [myid:127.0.0.1:30085] - INFO [main-SendThread(127.0.0.1:30085):ClientCnxnSocketNetty$ZKClientPipelineFactory@370] - SSL handler added for channel: null
2016-08-03 05:33:45,073 [myid:] - INFO [New I/O worker #221:ClientCnxn$SendThread@948] - Socket connection established, initiating session, client: /127.0.0.1:57010, server: 127.0.0.1/127.0.0.1:30085
2016-08-03 05:33:45,076 [myid:] - INFO [New I/O worker #221:ClientCnxnSocketNetty$1@153] - channel is connected: [id: 0x073f691d, /127.0.0.1:57010 => 127.0.0.1/127.0.0.1:30085]
2016-08-03 05:33:45,079 [myid:-1] - INFO [New I/O server boss #242:NettyServerCnxnFactory@384] - SSL handler added for channel: null
2016-08-03 05:33:45,214 [myid:-1] - INFO [New I/O worker #199:X509AuthenticationProvider@157] - Authenticated Id 'CN=localhost,OU=ZooKeeper,O=Apache,L=Unknown,ST=Unknown,C=Unknown' for Scheme 'x509'
2016-08-03 05:33:45,215 [myid:-1] - INFO [New I/O worker #199:ZooKeeperServer@964] - Client attempting to establish new session at /127.0.0.1:57010
2016-08-03 05:33:45,218 [myid:-1] - INFO [SyncThread:0:FileTxnLog@204] - Creating new log file: log.1
2016-08-03 05:33:45,256 [myid:2] - INFO [QuorumPeer[myid=2](plain=localhost/127.0.0.1:30081)(secure=0.0.0.0/0.0.0.0:30082):CommitProcessor@318] - Configuring CommitProcessor with 16 worker threads.
2016-08-03 05:33:45,260 [myid:2] - INFO [QuorumPeer[myid=2](plain=localhost/127.0.0.1:30081)(secure=0.0.0.0/0.0.0.0:30082):ContainerManager@64] - Using checkIntervalMs=60000 maxPerMinute=10000
2016-08-03 05:33:45,261 [myid:1] - INFO [QuorumPeer[myid=1](plain=localhost/127.0.0.1:30077)(secure=0.0.0.0/0.0.0.0:30078):Learner@493] - Learner received UPTODATE message
2016-08-03 05:33:45,261 [myid:0] - INFO [QuorumPeer[myid=0](plain=localhost/127.0.0.1:30073)(secure=0.0.0.0/0.0.0.0:30074):Learner@493] - Learner received UPTODATE message
2016-08-03 05:33:45,349 [myid:-1] - INFO [SyncThread:0:ZooKeeperServer@678] - Established session 0x101a7cee3fd0000 with negotiated timeout 8000 for client /127.0.0.1:57010
2016-08-03 05:33:45,350 [myid:] - INFO [New I/O worker #221:ClientCnxn$SendThread@1381] - Session establishment complete on server 127.0.0.1/127.0.0.1:30085, sessionid = 0x101a7cee3fd0000, negotiated timeout = 8000
2016-08-03 05:33:45,421 [myid:1] - INFO [QuorumPeer[myid=1](plain=localhost/127.0.0.1:30077)(secure=0.0.0.0/0.0.0.0:30078):CommitProcessor@318] - Configuring CommitProcessor with 16 worker threads.
2016-08-03 05:33:45,421 [myid:0] - INFO [QuorumPeer[myid=0](plain=localhost/127.0.0.1:30073)(secure=0.0.0.0/0.0.0.0:30074):CommitProcessor@318] - Configuring CommitProcessor with 16 worker threads.
2016-08-03 05:33:45,459 [myid:-1] - INFO [ProcessThread(sid:0 cport:-1)::PrepRequestProcessor@647] - Processed session termination for sessionid: 0x101a7cee3fd0000
2016-08-03 05:33:45,503 [myid:] - INFO [main:ClientCnxnSocketNetty@208] - channel is told closing
2016-08-03 05:33:45,503 [myid:] - INFO [main:ZooKeeper@1313] - Session: 0x101a7cee3fd0000 closed
2016-08-03 05:33:45,503 [myid:] - INFO [main-EventThread:ClientCnxn$EventThread@513] - EventThread shut down for session: 0x101a7cee3fd0000
2016-08-03 05:33:45,503 [myid:] - INFO [New I/O worker #221:ClientCnxnSocketNetty$ZKClientHandler@384] - channel is disconnected: [id: 0x073f691d, /127.0.0.1:57010 :> 127.0.0.1/127.0.0.1:30085]
2016-08-03 05:33:45,504 [myid:] - INFO [New I/O worker #221:ClientCnxnSocketNetty@208] - channel is told closing
2016-08-03 05:33:45,510 [myid:-1] - INFO [SyncThread:0:MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=StandaloneServer_port-1,name1=Connections,name2=127.0.0.1,name3=0x101a7cee3fd0000]
2016-08-03 05:33:46,003 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@82] - Memory used 92699
2016-08-03 05:33:46,004 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@87] - Number of threads 283
2016-08-03 05:33:46,004 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@102] - FINISHED TEST METHOD testSecureStandaloneServer
2016-08-03 05:33:46,004 [myid:] - INFO [main:ZKTestCase$1@65] - SUCCEEDED testSecureStandaloneServer
2016-08-03 05:33:46,004 [myid:] - INFO [main:ZKTestCase$1@60] - FINISHED testSecureStandaloneServer
{noformat}

flaky, flaky-test 9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
1 year, 21 weeks ago 0|i31u67:
ZooKeeper ZOOKEEPER-2498

Potential resource leak in C client when processing unexpected / out of order response

Bug Closed Major Fixed Michael Han Michael Han Michael Han 03/Aug/16 01:01   04/Sep/16 01:27 03/Aug/16 14:22 3.4.8, 3.5.2 3.4.9, 3.5.3 c client   0 4   In C client, we use reference counting to decide if a given zh handle can be destroyed or not. This requires we always make sure to call api_prolog (which increment the counter) and api_epilog (which decrease the counter) in pairs, for a given call context.

In zookeeper_process, there is a place where the code will return without invoking api_epilog, which would lead to potential zh resource leak.
9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
3 years, 33 weeks, 1 day ago 0|i31u5b:
ZooKeeper ZOOKEEPER-2497

ZOOKEEPER-3170 Flaky Test: org.apache.zookeeper.test.QuorumTest.testMultipleWatcherObjs

Sub-task Closed Major Cannot Reproduce Andor Molnar Michael Han Michael Han 02/Aug/16 11:27   19/Dec/19 18:02 25/Oct/18 11:06 3.5.2 3.5.5 quorum, tests   0 2   Example:
https://builds.apache.org/job/ZooKeeper-trunk-jdk8/607/
https://builds.apache.org/job/ZooKeeper_branch35_jdk8/127/

Note I haven't found any of the Jenkins JDK7 build fail with same error message, so not sure if this is JDK8 specific.

{noformat}
1 tests failed.
FAILED: org.apache.zookeeper.test.QuorumTest.testMultipleWatcherObjs

Error Message:
Timeout occurred. Please note the time in the report does not reflect the time until the timeout.

Stack Trace:
junit.framework.AssertionFailedError: Timeout occurred. Please note the time in the report does not reflect the time until the timeout.
at java.lang.Thread.run(Thread.java:745)
{noformat}
flaky, flaky-test 9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
1 year, 21 weeks ago 0|i31t3r:
ZooKeeper ZOOKEEPER-2496

When inside a transaction, some exceptions do not have path information set.

Bug Open Major Unresolved Unassigned Kazuaki Banzai Kazuaki Banzai 01/Aug/16 03:52   27/Sep/16 08:24   3.4.8, 3.5.1       1 6   If a client tries to execute some illegal operations inside a transaction, ZooKeeper throws an exception.

Some exceptions such as NodeExistsException should have a path to indicate where the exception occurred.
ZooKeeper clients can get the path by calling method getPath.
However, this method returns null if the exception occurs inside a transaction.
For example, when a client calls create /a and create /a in a transaction,
ZooKeeper throws NodeExistsException but getPath returns null.
In normal operation (outside transactions), the path information is set correctly.
The patch only shows this bug occurs with NoNode exception and NodeExists exception,
but this bug seems to occur with any exception which needs a path information:
When an error occurred in a transaction, ZooKeeper creates an ErrorResult instance to represent error result.

However, the ErrorResult class doesn't have a field for a path where an error occurred(See src/java/main/org/apache/zookeeper/OpResult.java for more details).
9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
3 years, 25 weeks, 2 days ago 0|i31qfr:
ZooKeeper ZOOKEEPER-2495

Cluster unavailable on disk full(ENOSPC), disk quota(EDQUOT), disk write error(EIO) errors

Bug Open Major Unresolved Unassigned Ramnatthan Alagappan Ramnatthan Alagappan 31/Jul/16 16:51   31/Jul/16 21:22   3.4.8   leaderElection, server   0 4   Normal ZooKeeper cluster with 3 Linux nodes. ZooKeeper cluster completely stalls with *no* transactions making progress when a storage related error (such as *ENOSPC, EDQUOT, EIO*) is encountered by the current *leader*.

Surprisingly, the same errors in some circumstances cause the node to completely crash and therefore allowing other nodes in the cluster to become the leader and make progress with transactions. Interestingly, the same errors if encountered while initializing a new log file causes the current leader to go to weird state (but does not crash) where it thinks it is the leader (and so does not allow others to become the leader). *This causes the entire cluster to freeze. *

Here is the stacktrace of the leader:

------------------------------------------------

2016-07-11 15:42:27,502 [myid:3] - INFO [SyncThread:3:FileTxnLog@199] - Creating new log file: log.200000001
2016-07-11 15:42:27,505 [myid:3] - ERROR [SyncThread:3:ZooKeeperCriticalThread@49] - Severe unrecoverable error, from thread : SyncThread:3
java.io.IOException: Disk quota exceeded
at java.io.FileOutputStream.writeBytes(Native Method)
at java.io.FileOutputStream.write(FileOutputStream.java:345)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
at org.apache.zookeeper.server.persistence.FileTxnLog.append(FileTxnLog.java:211)
at org.apache.zookeeper.server.persistence.FileTxnSnapLog.append(FileTxnSnapLog.java:314)
at org.apache.zookeeper.server.ZKDatabase.append(ZKDatabase.java:476)
at org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:140)

------------------------------------------------

From the trace and the code, it looks like the problem happens only when a new log file is initialized and only when there are errors in two cases:

1. Error during the append of *log header*.
2. Error during *padding zero bytes to the end of the log*.

If similar errors happen when writing some other blocks of data, then the node just completely crashes allowing others to be elected as a new leader. These two blocks of the newly created log file are special as they take a different error recovery code path -- the node does not completely crash but rather certain threads are killed but supposedly the quorum holding thread stays up thereby preventing others to become the new leader. This causes the other nodes to think that there is no problem with the leader but the cluster just becomes unavailable for any subsequent operations such as read/write.
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 33 weeks, 3 days ago 0|i31q2n:
ZooKeeper ZOOKEEPER-2494

reword documentation to say "a simple shell to execute file-like operations" instead of "a shell in which to execute simple file-system-like operations"

Bug Open Major Unresolved Abraham Fine Abraham Fine Abraham Fine 28/Jul/16 15:35   05/Feb/20 07:15   3.4.8, 3.5.2 3.7.0, 3.5.8 documentation   0 2   [~rakeshr] mades this suggestion in ZOOKEEPER-2477 and I thought this was worth implementing in its own jira 9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
2 years, 50 weeks ago 0|i31n5j:
ZooKeeper ZOOKEEPER-2493

ZOOKEEPER-3170 Flaky Tests: ReconfigTest fails on Solaris

Sub-task Open Major Unresolved Unassigned Michael Han Michael Han 28/Jul/16 13:34   05/Feb/20 07:17   3.5.0, 3.5.1, 3.5.2 3.7.0, 3.5.8 tests   0 1   Solaris Two tests related to ReconfigTest fail consistently on Solaris:

org.apache.zookeeper.test.ReconfigTest.testPortChangeToBlockedPortLeaderorg.apache.zookeeper.test.ReconfigTest.testPortChangeToBlockedPortFollower

Examples:
https://builds.apache.org/view/S-Z/view/ZooKeeper/job/ZooKeeper-trunk-solaris/1245/#showFailuresLink
https://builds.apache.org/view/S-Z/view/ZooKeeper/job/ZooKeeper_branch35_solaris/185/#showFailuresLink

{noformat}
org.apache.zookeeper.test.ReconfigTest.testPortChangeToBlockedPortFollower

Failing for the past 5 builds (Since Failed#1241 )
Took 1 min 37 sec.
Error Message

client could not connect to reestablished quorum: giving up after 30+ seconds.
Stacktrace

junit.framework.AssertionFailedError: client could not connect to reestablished quorum: giving up after 30+ seconds.
at org.apache.zookeeper.test.ReconfigTest.testNormalOperation(ReconfigTest.java:173)
at org.apache.zookeeper.test.ReconfigTest.testPortChangeToBlockedPort(ReconfigTest.java:732)
at org.apache.zookeeper.test.ReconfigTest.testPortChangeToBlockedPortFollower(ReconfigTest.java:658)
at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:79)
Standard Output

2016-07-28 09:24:37,121 [myid:] - INFO [main:JUnit4ZKTestRunner@47] - No test.method specified. using default methods.
2016-07-28 09:24:37,185 [myid:] - INFO [main:JUnit4ZKTestRunner@47] - No test.method specified. using default methods.
2016-07-28 09:24:37,200 [myid:] - INFO [main:ZKTestCase$1@55] - STARTING testQuorumSystemChange
2016-07-28 09:24:37,202 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@77] - RUNNING TEST METHOD testQuorumSystemChange
2016-07-28 09:24:37,577 [myid:] - INFO [main:PortAssignment@157] - Single test process using ports from 11221 - 32767.
2016-07-28 09:24:37,577 [myid:] - INFO [main:PortAssignment@85] - Assigned port 11222 from range 11221 - 32767.
2016-07-28 09:24:37,579 [myid:] - INFO [main:PortAssignment@85] - Assigned port 11223 from range 11221 - 32767.
2016-07-28 09:24:37,579 [myid:] - INFO [main:PortAssignment@85] - Assigned port 11224 from range 11221 - 32767.
2016-07-28 09:24:37,580 [myid:] - INFO [main:PortAssignment@85] - Assigned port 11225 from range 11221 - 32767.
2016-07-28 09:24:37,581 [myid:] - INFO [main:PortAssignment@85] - Assigned port 11226 from range 11221 - 32767.
2016-07-28 09:24:37,581 [myid:] - INFO [main:PortAssignment@85] - Assigned port 11227 from range 11221 - 32767.
2016-07-28 09:24:37,581 [myid:] - INFO [main:PortAssignment@85] - Assigned port 11228 from range 11221 - 32767.
2016-07-28 09:24:37,582 [myid:] - INFO [main:PortAssignment@85] - Assigned port 11229 from range 11221 - 32767.
2016-07-28 09:24:37,582 [myid:] - INFO [main:PortAssignment@85] - Assigned port 11230 from range 11221 - 32767.
2016-07-28 09:24:37,582 [myid:] - INFO [main:PortAssignment@85] - Assigned port 11231 from range 11221 - 32767.
2016-07-28 09:24:37,582 [myid:] - INFO [main:PortAssignment@85] - Assigned port 11232 from range 11221 - 32767.
2016-07-28 09:24:37,584 [myid:] - INFO [main:PortAssignment@85] - Assigned port 11233 from range 11221 - 32767.
2016-07-28 09:24:37,585 [myid:] - INFO [main:PortAssignment@85] - Assigned port 11234 from range 11221 - 32767.
2016-07-28 09:24:37,585 [myid:] - INFO [main:PortAssignment@85] - Assigned port 11235 from range 11221 - 32767.
2016-07-28 09:24:37,585 [myid:] - INFO [main:PortAssignment@85] - Assigned port 11236 from range 11221 - 32767.
2016-07-28 09:24:37,585 [myid:] - INFO [main:PortAssignment@85] - Assigned port 11237 from range 11221 - 32767.
2016-07-28 09:24:37,586 [myid:] - INFO [main:PortAssignment@85] - Assigned port 11238 from range 11221 - 32767.
2016-07-28 09:24:37,586 [myid:] - INFO [main:PortAssignment@85] - Assigned port 11239 from range 11221 - 32767.
2016-07-28 09:24:37,586 [myid:] - INFO [main:PortAssignment@85] - Assigned port 11240 from range 11221 - 32767.
2016-07-28 09:24:37,587 [myid:] - INFO [main:PortAssignment@85] - Assigned port 11241 from range 11221 - 32767.
2016-07-28 09:24:37,587 [myid:] - INFO [main:PortAssignment@85] - Assigned port 11242 from range 11221 - 32767.
2016-07-28 09:24:37,587 [myid:] - INFO [main:QuorumUtil@116] - Creating QuorumPeer 1; public port 11222
2016-07-28 09:24:37,609 [myid:] - INFO [main:NIOServerCnxnFactory@673] - Configuring NIO connection handler with 10s sessionless connection timeout, 2 selector thread(s), 16 worker threads, and 64 kB direct buffers.
2016-07-28 09:24:37,615 [myid:] - INFO [main:NIOServerCnxnFactory@686] - binding to port /127.0.0.1:11222
2016-07-28 09:24:37,638 [myid:] - INFO [main:QuorumUtil@116] - Creating QuorumPeer 2; public port 11225
2016-07-28 09:24:37,638 [myid:] - INFO [main:NIOServerCnxnFactory@673] - Configuring NIO connection handler with 10s sessionless connection timeout, 2 selector thread(s), 16 worker threads, and 64 kB direct buffers.
2016-07-28 09:24:37,639 [myid:] - INFO [main:NIOServerCnxnFactory@686] - binding to port /127.0.0.1:11225
2016-07-28 09:24:37,639 [myid:] - INFO [main:QuorumUtil@116] - Creating QuorumPeer 3; public port 11228
2016-07-28 09:24:37,640 [myid:] - INFO [main:NIOServerCnxnFactory@673] - Configuring NIO connection handler with 10s sessionless connection timeout, 2 selector thread(s), 16 worker threads, and 64 kB direct buffers.
2016-07-28 09:24:37,640 [myid:] - INFO [main:NIOServerCnxnFactory@686] - binding to port /127.0.0.1:11228
2016-07-28 09:24:37,641 [myid:] - INFO [main:QuorumUtil@116] - Creating QuorumPeer 4; public port 11231
2016-07-28 09:24:37,641 [myid:] - INFO [main:NIOServerCnxnFactory@673] - Configuring NIO connection handler with 10s sessionless connection timeout, 2 selector thread(s), 16 worker threads, and 64 kB direct buffers.
2016-07-28 09:24:37,641 [myid:] - INFO [main:NIOServerCnxnFactory@686] - binding to port /127.0.0.1:11231
2016-07-28 09:24:37,642 [myid:] - INFO [main:QuorumUtil@116] - Creating QuorumPeer 5; public port 11234
2016-07-28 09:24:37,642 [myid:] - INFO [main:NIOServerCnxnFactory@673] - Configuring NIO connection handler with 10s sessionless connection timeout, 2 selector thread(s), 16 worker threads, and 64 kB direct buffers.
2016-07-28 09:24:37,642 [myid:] - INFO [main:NIOServerCnxnFactory@686] - binding to port /127.0.0.1:11234
2016-07-28 09:24:37,643 [myid:] - INFO [main:QuorumUtil@116] - Creating QuorumPeer 6; public port 11237
2016-07-28 09:24:37,643 [myid:] - INFO [main:NIOServerCnxnFactory@673] - Configuring NIO connection handler with 10s sessionless connection timeout, 2 selector thread(s), 16 worker threads, and 64 kB direct buffers.
2016-07-28 09:24:37,644 [myid:] - INFO [main:NIOServerCnxnFactory@686] - binding to port /127.0.0.1:11237
2016-07-28 09:24:37,644 [myid:] - INFO [main:QuorumUtil@116] - Creating QuorumPeer 7; public port 11240
2016-07-28 09:24:37,644 [myid:] - INFO [main:NIOServerCnxnFactory@673] - Configuring NIO connection handler with 10s sessionless connection timeout, 2 selector thread(s), 16 worker threads, and 64 kB direct buffers.
2016-07-28 09:24:37,645 [myid:] - INFO [main:NIOServerCnxnFactory@686] - binding to port /127.0.0.1:11240
2016-07-28 09:24:37,646 [myid:] - INFO [main:QuorumUtil@250] - Shutting down quorum peer QuorumPeer
2016-07-28 09:24:37,646 [myid:] - INFO [main:QuorumUtil@257] - No election available to shutdown QuorumPeer
2016-07-28 09:24:37,646 [myid:] - INFO [main:QuorumUtil@259] - Waiting for QuorumPeer to exit thread
2016-07-28 09:24:37,646 [myid:] - INFO [main:QuorumUtil@250] - Shutting down quorum peer QuorumPeer
2016-07-28 09:24:37,647 [myid:] - INFO [main:QuorumUtil@257] - No election available to shutdown QuorumPeer
2016-07-28 09:24:37,647 [myid:] - INFO [main:QuorumUtil@259] - Waiting for QuorumPeer to exit thread
2016-07-28 09:24:37,647 [myid:] - INFO [main:QuorumUtil@250] - Shutting down quorum peer QuorumPeer
2016-07-28 09:24:37,647 [myid:] - INFO [main:QuorumUtil@257] - No election available to shutdown QuorumPeer
2016-07-28 09:24:37,647 [myid:] - INFO [main:QuorumUtil@259] - Waiting for QuorumPeer to exit thread
2016-07-28 09:24:37,647 [myid:] - INFO [main:QuorumUtil@250] - Shutting down quorum peer QuorumPeer
2016-07-28 09:24:37,647 [myid:] - INFO [main:QuorumUtil@257] - No election available to shutdown QuorumPeer
2016-07-28 09:24:37,647 [myid:] - INFO [main:QuorumUtil@259] - Waiting for QuorumPeer to exit thread
2016-07-28 09:24:37,648 [myid:] - INFO [main:QuorumUtil@250] - Shutting down quorum peer QuorumPeer
2016-07-28 09:24:37,648 [myid:] - INFO [main:QuorumUtil@257] - No election available to shutdown QuorumPeer
2016-07-28 09:24:37,648 [myid:] - INFO [main:QuorumUtil@259] - Waiting for QuorumPeer to exit thread
2016-07-28 09:24:37,648 [myid:] - INFO [main:QuorumUtil@250] - Shutting down quorum peer QuorumPeer
2016-07-28 09:24:37,648 [myid:] - INFO [main:QuorumUtil@257] - No election available to shutdown QuorumPeer
2016-07-28 09:24:37,648 [myid:] - INFO [main:QuorumUtil@259] - Waiting for QuorumPeer to exit thread
2016-07-28 09:24:37,648 [myid:] - INFO [main:QuorumUtil@250] - Shutting down quorum peer QuorumPeer
2016-07-28 09:24:37,649 [myid:] - INFO [main:QuorumUtil@257] - No election available to shutdown QuorumPeer
2016-07-28 09:24:37,649 [myid:] - INFO [main:QuorumUtil@259] - Waiting for QuorumPeer to exit thread
2016-07-28 09:24:37,651 [myid:] - INFO [main:FourLetterWordMain@85] - connecting to 127.0.0.1 11222
2016-07-28 09:24:37,651 [myid:] - INFO [main:QuorumUtil@243] - 127.0.0.1:11222 is no longer accepting client connections
2016-07-28 09:24:37,651 [myid:] - INFO [main:FourLetterWordMain@85] - connecting to 127.0.0.1 11225
2016-07-28 09:24:37,652 [myid:] - INFO [main:QuorumUtil@243] - 127.0.0.1:11225 is no longer accepting client connections
2016-07-28 09:24:37,652 [myid:] - INFO [main:FourLetterWordMain@85] - connecting to 127.0.0.1 11228
2016-07-28 09:24:37,652 [myid:] - INFO [main:QuorumUtil@243] - 127.0.0.1:11228 is no longer accepting client connections
2016-07-28 09:24:37,652 [myid:] - INFO [main:FourLetterWordMain@85] - connecting to 127.0.0.1 11231
2016-07-28 09:24:37,653 [myid:] - INFO [main:QuorumUtil@243] - 127.0.0.1:11231 is no longer accepting client connections
2016-07-28 09:24:37,653 [myid:] - INFO [main:FourLetterWordMain@85] - connecting to 127.0.0.1 11234
2016-07-28 09:24:37,653 [myid:] - INFO [main:QuorumUtil@243] - 127.0.0.1:11234 is no longer accepting client connections
2016-07-28 09:24:37,653 [myid:] - INFO [main:FourLetterWordMain@85] - connecting to 127.0.0.1 11237
2016-07-28 09:24:37,653 [myid:] - INFO [main:QuorumUtil@243] - 127.0.0.1:11237 is no longer accepting client connections
2016-07-28 09:24:37,654 [myid:] - INFO [main:FourLetterWordMain@85] - connecting to 127.0.0.1 11240
2016-07-28 09:24:37,654 [myid:] - INFO [main:QuorumUtil@243] - 127.0.0.1:11240 is no longer accepting client connections
2016-07-28 09:24:37,654 [myid:] - INFO [main:QuorumUtil@203] - Creating QuorumPeer 1; public port 11222
2016-07-28 09:24:37,654 [myid:] - INFO [main:NIOServerCnxnFactory@673] - Configuring NIO connection handler with 10s sessionless connection timeout, 2 selector thread(s), 16 worker threads, and 64 kB direct buffers.
2016-07-28 09:24:37,655 [myid:] - INFO [main:NIOServerCnxnFactory@686] - binding to port /127.0.0.1:11222
2016-07-28 09:24:37,660 [myid:] - INFO [main:QuorumPeer@776] - currentEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
2016-07-28 09:24:37,708 [myid:] - INFO [main:QuorumPeer@791] - acceptedEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
2016-07-28 09:24:37,754 [myid:] - INFO [main:QuorumUtil@146] - Started QuorumPeer 1
2016-07-28 09:24:37,754 [myid:] - INFO [main:QuorumUtil@203] - Creating QuorumPeer 2; public port 11225
2016-07-28 09:24:37,755 [myid:] - INFO [main:NIOServerCnxnFactory@673] - Configuring NIO connection handler with 10s sessionless connection timeout, 2 selector thread(s), 16 worker threads, and 64 kB direct buffers.
2016-07-28 09:24:37,755 [myid:] - INFO [main:NIOServerCnxnFactory@686] - binding to port /127.0.0.1:11225
2016-07-28 09:24:37,756 [myid:] - INFO [main:QuorumPeer@776] - currentEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
2016-07-28 09:24:37,759 [myid:] - INFO [QuorumPeerListener:QuorumCnxManager$Listener@632] - My election bind port: /127.0.0.1:11224
2016-07-28 09:24:37,771 [myid:] - INFO [main:QuorumPeer@791] - acceptedEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
2016-07-28 09:24:37,780 [myid:] - INFO [main:QuorumUtil@146] - Started QuorumPeer 2
2016-07-28 09:24:37,780 [myid:] - INFO [main:QuorumUtil@203] - Creating QuorumPeer 3; public port 11228
2016-07-28 09:24:37,781 [myid:] - INFO [main:NIOServerCnxnFactory@673] - Configuring NIO connection handler with 10s sessionless connection timeout, 2 selector thread(s), 16 worker threads, and 64 kB direct buffers.
2016-07-28 09:24:37,781 [myid:] - INFO [main:NIOServerCnxnFactory@686] - binding to port /127.0.0.1:11228
2016-07-28 09:24:37,781 [myid:] - INFO [QuorumPeerListener:QuorumCnxManager$Listener@632] - My election bind port: /127.0.0.1:11227
2016-07-28 09:24:37,888 [myid:] - INFO [main:QuorumPeer@776] - currentEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
2016-07-28 09:24:37,894 [myid:] - INFO [QuorumPeer[myid=1](plain=/127.0.0.1:11222)(secure=disabled):QuorumPeer@1033] - LOOKING
2016-07-28 09:24:37,894 [myid:] - INFO [QuorumPeer[myid=2](plain=/127.0.0.1:11225)(secure=disabled):QuorumPeer@1033] - LOOKING
2016-07-28 09:24:37,895 [myid:] - INFO [QuorumPeer[myid=2](plain=/127.0.0.1:11225)(secure=disabled):FastLeaderElection@894] - New election. My id = 2, proposed zxid=0x0
2016-07-28 09:24:37,895 [myid:] - INFO [QuorumPeer[myid=1](plain=/127.0.0.1:11222)(secure=disabled):FastLeaderElection@894] - New election. My id = 1, proposed zxid=0x0
2016-07-28 09:24:37,899 [myid:] - INFO [/127.0.0.1:11227:QuorumCnxManager$Listener@638] - Received connection request /127.0.0.1:50183
2016-07-28 09:24:37,899 [myid:] - INFO [WorkerSender[myid=1]:QuorumCnxManager@276] - Have smaller server identifier, so dropping the connection: (2, 1)
2016-07-28 09:24:37,901 [myid:] - INFO [/127.0.0.1:11224:QuorumCnxManager$Listener@638] - Received connection request /127.0.0.1:50185
2016-07-28 09:24:37,901 [myid:] - WARN [WorkerSender[myid=1]:QuorumCnxManager@455] - Cannot open channel to 3 at election address /127.0.0.1:11230
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:441)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:482)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:419)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:486)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:465)
at java.lang.Thread.run(Thread.java:745)
2016-07-28 09:24:37,905 [myid:] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@688] - Notification: 2 (message format version), 1 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version)
2016-07-28 09:24:37,906 [myid:] - WARN [WorkerSender[myid=2]:QuorumCnxManager@455] - Cannot open channel to 3 at election address /127.0.0.1:11230
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:441)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:482)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:419)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:486)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:465)
at java.lang.Thread.run(Thread.java:745)
2016-07-28 09:24:37,906 [myid:] - WARN [WorkerSender[myid=2]:QuorumCnxManager@455] - Cannot open channel to 4 at election address /127.0.0.1:11233
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:441)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:482)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:419)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:486)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:465)
at java.lang.Thread.run(Thread.java:745)
2016-07-28 09:24:37,905 [myid:] - WARN [WorkerSender[myid=1]:QuorumCnxManager@455] - Cannot open channel to 4 at election address /127.0.0.1:11233
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:441)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:482)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:419)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:486)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:465)
at java.lang.Thread.run(Thread.java:745)
2016-07-28 09:24:37,907 [myid:] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@688] - Notification: 2 (message format version), 1 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version)
2016-07-28 09:24:37,908 [myid:] - WARN [WorkerSender[myid=1]:QuorumCnxManager@455] - Cannot open channel to 5 at election address /127.0.0.1:11236
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:441)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:482)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:419)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:486)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:465)
at java.lang.Thread.run(Thread.java:745)
2016-07-28 09:24:37,920 [myid:] - INFO [main:QuorumPeer@791] - acceptedEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
2016-07-28 09:24:37,961 [myid:] - WARN [WorkerSender[myid=2]:QuorumCnxManager@455] - Cannot open channel to 5 at election address /127.0.0.1:11236
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:441)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:482)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:419)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:486)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:465)
at java.lang.Thread.run(Thread.java:745)
2016-07-28 09:24:37,908 [myid:] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@688] - Notification: 2 (message format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version)
2016-07-28 09:24:37,962 [myid:] - WARN [WorkerSender[myid=1]:QuorumCnxManager@455] - Cannot open channel to 6 at election address /127.0.0.1:11239
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:441)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:482)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:419)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:486)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:465)
at java.lang.Thread.run(Thread.java:745)
2016-07-28 09:24:37,962 [myid:] - WARN [WorkerSender[myid=2]:QuorumCnxManager@455] - Cannot open channel to 6 at election address /127.0.0.1:11239
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:441)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:482)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:419)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:486)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:465)
at java.lang.Thread.run(Thread.java:745)
2016-07-28 09:24:37,963 [myid:] - WARN [WorkerSender[myid=1]:QuorumCnxManager@455] - Cannot open channel to 7 at election address /127.0.0.1:11242
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:441)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:482)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:419)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:486)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:465)
at java.lang.Thread.run(Thread.java:745)
2016-07-28 09:24:37,963 [myid:] - WARN [WorkerSender[myid=2]:QuorumCnxManager@455] - Cannot open channel to 7 at election address /127.0.0.1:11242
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:441)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:482)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:419)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:486)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:465)
at java.lang.Thread.run(Thread.java:745)
2016-07-28 09:24:37,965 [myid:] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@688] - Notification: 2 (message format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version)
2016-07-28 09:24:37,967 [myid:] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@688] - Notification: 2 (message format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version)
2016-07-28 09:24:37,968 [myid:] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@688] - Notification: 2 (message format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version)
2016-07-28 09:24:37,968 [myid:] - WARN [WorkerSender[myid=1]:QuorumCnxManager@455] - Cannot open channel to 3 at election address /127.0.0.1:11230
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:441)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:482)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:419)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:486)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:465)
at java.lang.Thread.run(Thread.java:745)
2016-07-28 09:24:37,969 [myid:] - WARN [WorkerSender[myid=1]:QuorumCnxManager@455] - Cannot open channel to 4 at election address /127.0.0.1:11233
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:441)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:482)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:419)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:486)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:465)
at java.lang.Thread.run(Thread.java:745)
2016-07-28 09:24:37,969 [myid:] - WARN [WorkerSender[myid=1]:QuorumCnxManager@455] - Cannot open channel to 5 at election address /127.0.0.1:11236
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:441)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:482)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:419)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:486)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:465)
at java.lang.Thread.run(Thread.java:745)
2016-07-28 09:24:37,981 [myid:] - WARN [WorkerSender[myid=1]:QuorumCnxManager@455] - Cannot open channel to 6 at election address /127.0.0.1:11239
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:441)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:482)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:419)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:486)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:465)
at java.lang.Thread.run(Thread.java:745)
2016-07-28 09:24:37,982 [myid:] - WARN [WorkerSender[myid=1]:QuorumCnxManager@455] - Cannot open channel to 7 at election address /127.0.0.1:11242
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:441)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:482)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:419)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:486)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:465)
at java.lang.Thread.run(Thread.java:745)
2016-07-28 09:24:38,015 [myid:] - INFO [main:QuorumUtil@146] - Started QuorumPeer 3
2016-07-28 09:24:38,016 [myid:] - INFO [main:QuorumUtil@203] - Creating QuorumPeer 4; public port 11231
2016-07-28 09:24:38,016 [myid:] - INFO [main:NIOServerCnxnFactory@673] - Configuring NIO connection handler with 10s sessionless connection timeout, 2 selector thread(s), 16 worker threads, and 64 kB direct buffers.
2016-07-28 09:24:38,017 [myid:] - INFO [main:NIOServerCnxnFactory@686] - binding to port /127.0.0.1:11231
2016-07-28 09:24:38,017 [myid:] - INFO [QuorumPeer[myid=3](plain=/127.0.0.1:11228)(secure=disabled):QuorumPeer@1033] - LOOKING
2016-07-28 09:24:38,018 [myid:] - INFO [QuorumPeer[myid=3](plain=/127.0.0.1:11228)(secure=disabled):FastLeaderElection@894] - New election. My id = 3, proposed zxid=0x0
2016-07-28 09:24:38,018 [myid:] - INFO [main:QuorumPeer@776] - currentEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
2016-07-28 09:24:38,020 [myid:] - INFO [QuorumPeerListener:QuorumCnxManager$Listener@632] - My election bind port: /127.0.0.1:11230
2016-07-28 09:24:38,025 [myid:] - INFO [/127.0.0.1:11224:QuorumCnxManager$Listener@638] - Received connection request /127.0.0.1:50200
2016-07-28 09:24:38,026 [myid:] - INFO [/127.0.0.1:11227:QuorumCnxManager$Listener@638] - Received connection request /127.0.0.1:50201
2016-07-28 09:24:38,027 [myid:] - WARN [WorkerSender[myid=3]:QuorumCnxManager@455] - Cannot open channel to 4 at election address /127.0.0.1:11233
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:441)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:482)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:419)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:486)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:465)
at java.lang.Thread.run(Thread.java:745)
2016-07-28 09:24:38,027 [myid:] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@688] - Notification: 2 (message format version), 3 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 3 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version)
2016-07-28 09:24:38,028 [myid:] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@688] - Notification: 2 (message format version), 3 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 3 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version)
2016-07-28 09:24:38,029 [myid:] - INFO [WorkerReceiver[myid=3]:FastLeaderElection@688] - Notification: 2 (message format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version)
2016-07-28 09:24:38,029 [myid:] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@688] - Notification: 2 (message format version), 3 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version)
2016-07-28 09:24:38,031 [myid:] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@688] - Notification: 2 (message format version), 3 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version)
2016-07-28 09:24:38,031 [myid:] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@688] - Notification: 2 (message format version), 3 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version)
2016-07-28 09:24:38,031 [myid:] - INFO [WorkerReceiver[myid=3]:FastLeaderElection@688] - Notification: 2 (message format version), 3 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 3 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version)
2016-07-28 09:24:38,032 [myid:] - WARN [WorkerSender[myid=1]:QuorumCnxManager@455] - Cannot open channel to 4 at election address /127.0.0.1:11233
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:441)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:482)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:419)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:486)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:465)
at java.lang.Thread.run(Thread.java:745)
2016-07-28 09:24:38,032 [myid:] - WARN [WorkerSender[myid=1]:QuorumCnxManager@455] - Cannot open channel to 5 at election address /127.0.0.1:11236
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:441)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:482)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:419)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:486)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:465)
at java.lang.Thread.run(Thread.java:745)
2016-07-28 09:24:38,033 [myid:] - WARN [WorkerSender[myid=1]:QuorumCnxManager@455] - Cannot open channel to 6 at election address /127.0.0.1:11239
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:441)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:482)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:419)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:486)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:465)
at java.lang.Thread.run(Thread.java:745)
2016-07-28 09:24:38,033 [myid:] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@688] - Notification: 2 (message format version), 3 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version)
2016-07-28 09:24:38,032 [myid:] - INFO [WorkerReceiver[myid=3]:FastLeaderElection@688] - Notification: 2 (message format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version)
2016-07-28 09:24:38,033 [myid:] - WARN [WorkerSender[myid=2]:QuorumCnxManager@455] - Cannot open channel to 4 at election address /127.0.0.1:11233
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:441)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:482)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:419)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:486)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:465)
at java.lang.Thread.run(Thread.java:745)
2016-07-28 09:24:38,034 [myid:] - WARN [WorkerSender[myid=1]:QuorumCnxManager@455] - Cannot open channel to 7 at election address /127.0.0.1:11242
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:441)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:482)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:419)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:486)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:465)
at java.lang.Thread.run(Thread.java:745)
2016-07-28 09:24:38,035 [myid:] - INFO [WorkerReceiver[myid=3]:FastLeaderElection@688] - Notification: 2 (message format version), 3 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version)
2016-07-28 09:24:38,036 [myid:] - WARN [WorkerSender[myid=2]:QuorumCnxManager@455] - Cannot open channel to 5 at election address /127.0.0.1:11236
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:441)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:482)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:419)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:486)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:465)
at java.lang.Thread.run(Thread.java:745)
2016-07-28 09:24:38,037 [myid:] - INFO [WorkerReceiver[myid=3]:FastLeaderElection@688] - Notification: 2 (message format version), 3 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version)
2016-07-28 09:24:38,037 [myid:] - WARN [WorkerSender[myid=2]:QuorumCnxManager@455] - Cannot open channel to 6 at election address /127.0.0.1:11239
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:441)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:482)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:419)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:486)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:465)
at java.lang.Thread.run(Thread.java:745)
2016-07-28 09:24:38,038 [myid:] - WARN [WorkerSender[myid=2]:QuorumCnxManager@455] - Cannot open channel to 7 at election address /127.0.0.1:11242
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:441)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:482)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:419)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:486)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:465)
at java.lang.Thread.run(Thread.java:745)
2016-07-28 09:24:38,044 [myid:] - WARN [WorkerSender[myid=3]:QuorumCnxManager@455] - Cannot open channel to 5 at election address /127.0.0.1:11236
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:441)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:482)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:419)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:486)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:465)
at java.lang.Thread.run(Thread.java:745)
2016-07-28 09:24:38,045 [myid:] - WARN [WorkerSender[myid=3]:QuorumCnxManager@455] - Cannot open channel to 6 at election address /127.0.0.1:11239
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:441)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:482)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:419)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:486)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:465)
at java.lang.Thread.run(Thread.java:745)
2016-07-28 09:24:38,045 [myid:] - WARN [WorkerSender[myid=3]:QuorumCnxManager@455] - Cannot open channel to 7 at election address /127.0.0.1:11242
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)

...[truncated 3451538 chars]...
mPeer[myid=1](plain=/127.0.0.1:11371)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id1]
2016-07-28 09:30:21,332 [myid:] - INFO [QuorumPeer[myid=1](plain=/127.0.0.1:11371)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id1,name1=replica.1]
2016-07-28 09:30:21,332 [myid:] - INFO [QuorumPeer[myid=1](plain=/127.0.0.1:11371)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id1,name1=replica.2]
2016-07-28 09:30:21,332 [myid:] - INFO [QuorumPeer[myid=1](plain=/127.0.0.1:11371)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id1,name1=replica.3]
2016-07-28 09:30:21,332 [myid:] - INFO [QuorumPeer[myid=1](plain=/127.0.0.1:11371)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id1,name1=replica.5]
2016-07-28 09:30:21,332 [myid:] - INFO [main:QuorumUtil@250] - Shutting down quorum peer QuorumPeer[myid=2](plain=/127.0.0.1:11374)(secure=disabled)
2016-07-28 09:30:21,332 [myid:] - INFO [main:Follower@198] - shutdown called
java.lang.Exception: shutdown Follower
at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:198)
at org.apache.zookeeper.server.quorum.QuorumPeer.shutdown(QuorumPeer.java:1184)
at org.apache.zookeeper.test.QuorumUtil.shutdown(QuorumUtil.java:251)
at org.apache.zookeeper.test.QuorumUtil.shutdownAll(QuorumUtil.java:238)
at org.apache.zookeeper.test.QuorumUtil.tearDown(QuorumUtil.java:306)
at org.apache.zookeeper.test.ReconfigTest.tearDown(ReconfigTest.java:64)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33)
at org.junit.rules.TestWatchman$1.evaluate(TestWatchman.java:53)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:38)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:518)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1052)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:906)
2016-07-28 09:30:21,333 [myid:] - INFO [main:LearnerZooKeeperServer@165] - Shutting down
2016-07-28 09:30:21,333 [myid:] - INFO [main:ZooKeeperServer@498] - shutting down
2016-07-28 09:30:21,332 [myid:] - INFO [QuorumPeer[myid=2](plain=/127.0.0.1:11374)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id2,name1=replica.2,name2=Follower]
2016-07-28 09:30:21,333 [myid:] - INFO [main:FollowerRequestProcessor@138] - Shutting down
2016-07-28 09:30:21,333 [myid:] - INFO [QuorumPeer[myid=2](plain=/127.0.0.1:11374)(secure=disabled):Follower@198] - shutdown called
java.lang.Exception: shutdown Follower
at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:198)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1115)
2016-07-28 09:30:21,333 [myid:] - INFO [FollowerRequestProcessor:2:FollowerRequestProcessor@109] - FollowerRequestProcessor exited loop!
2016-07-28 09:30:21,333 [myid:] - INFO [main:CommitProcessor@414] - Shutting down
2016-07-28 09:30:21,334 [myid:] - INFO [CommitProcessor:2:CommitProcessor@299] - CommitProcessor exited loop!
2016-07-28 09:30:21,334 [myid:] - INFO [main:FinalRequestProcessor@479] - shutdown of request processor complete
2016-07-28 09:30:21,334 [myid:] - INFO [main:MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id2,name1=replica.2,name2=Follower,name3=InMemoryDataTree]
2016-07-28 09:30:21,334 [myid:] - INFO [main:SyncRequestProcessor@191] - Shutting down
2016-07-28 09:30:21,334 [myid:] - INFO [SyncThread:2:SyncRequestProcessor@169] - SyncRequestProcessor exited!
2016-07-28 09:30:21,334 [myid:] - WARN [QuorumPeer[myid=2](plain=/127.0.0.1:11374)(secure=disabled):QuorumPeer@1158] - PeerState set to LOOKING
2016-07-28 09:30:21,335 [myid:] - WARN [QuorumPeer[myid=2](plain=/127.0.0.1:11374)(secure=disabled):QuorumPeer@1140] - QuorumPeer main thread exited
2016-07-28 09:30:21,335 [myid:] - INFO [QuorumPeer[myid=2](plain=/127.0.0.1:11374)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id2]
2016-07-28 09:30:21,335 [myid:] - INFO [ConnnectionExpirer:NIOServerCnxnFactory$ConnectionExpirerThread@583] - ConnnectionExpirerThread interrupted
2016-07-28 09:30:21,335 [myid:] - INFO [QuorumPeer[myid=2](plain=/127.0.0.1:11374)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id2,name1=replica.2]
2016-07-28 09:30:21,336 [myid:] - INFO [NIOServerCxnFactory.AcceptThread:/127.0.0.1:11374:NIOServerCnxnFactory$AcceptThread@219] - accept thread exitted run method
2016-07-28 09:30:21,337 [myid:] - INFO [QuorumPeer[myid=2](plain=/127.0.0.1:11374)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id2,name1=replica.1]
2016-07-28 09:30:21,337 [myid:] - INFO [QuorumPeer[myid=2](plain=/127.0.0.1:11374)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id2,name1=replica.3]
2016-07-28 09:30:21,337 [myid:] - INFO [NIOServerCxnFactory.SelectorThread-0:NIOServerCnxnFactory$SelectorThread@420] - selector thread exitted run method
2016-07-28 09:30:21,337 [myid:] - INFO [QuorumPeer[myid=2](plain=/127.0.0.1:11374)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id2,name1=replica.5]
2016-07-28 09:30:21,337 [myid:] - INFO [NIOServerCxnFactory.SelectorThread-1:NIOServerCnxnFactory$SelectorThread@420] - selector thread exitted run method
2016-07-28 09:30:21,337 [myid:] - INFO [/127.0.0.1:11376:QuorumCnxManager$Listener@661] - Leaving listener
2016-07-28 09:30:21,338 [myid:] - INFO [main:QuorumUtil@254] - Shutting down leader election QuorumPeer[myid=2](plain=/127.0.0.1:11374)(secure=disabled)
2016-07-28 09:30:21,339 [myid:] - INFO [main:QuorumUtil@259] - Waiting for QuorumPeer[myid=2](plain=/127.0.0.1:11374)(secure=disabled) to exit thread
2016-07-28 09:30:21,339 [myid:] - INFO [main:QuorumUtil@250] - Shutting down quorum peer QuorumPeer[myid=3](plain=/127.0.0.1:11377)(secure=disabled)
2016-07-28 09:30:21,339 [myid:] - INFO [main:Follower@198] - shutdown called
java.lang.Exception: shutdown Follower
at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:198)
at org.apache.zookeeper.server.quorum.QuorumPeer.shutdown(QuorumPeer.java:1184)
at org.apache.zookeeper.test.QuorumUtil.shutdown(QuorumUtil.java:251)
at org.apache.zookeeper.test.QuorumUtil.shutdownAll(QuorumUtil.java:238)
at org.apache.zookeeper.test.QuorumUtil.tearDown(QuorumUtil.java:306)
at org.apache.zookeeper.test.ReconfigTest.tearDown(ReconfigTest.java:64)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33)
at org.junit.rules.TestWatchman$1.evaluate(TestWatchman.java:53)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:38)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:518)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1052)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:906)
2016-07-28 09:30:21,339 [myid:] - INFO [main:LearnerZooKeeperServer@165] - Shutting down
2016-07-28 09:30:21,339 [myid:] - INFO [main:ZooKeeperServer@498] - shutting down
2016-07-28 09:30:21,339 [myid:] - INFO [main:FollowerRequestProcessor@138] - Shutting down
2016-07-28 09:30:21,339 [myid:] - INFO [main:CommitProcessor@414] - Shutting down
2016-07-28 09:30:21,339 [myid:] - INFO [FollowerRequestProcessor:3:FollowerRequestProcessor@109] - FollowerRequestProcessor exited loop!
2016-07-28 09:30:21,339 [myid:] - INFO [CommitProcessor:3:CommitProcessor@299] - CommitProcessor exited loop!
2016-07-28 09:30:21,339 [myid:] - INFO [main:FinalRequestProcessor@479] - shutdown of request processor complete
2016-07-28 09:30:21,340 [myid:] - INFO [main:MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id3,name1=replica.3,name2=Follower,name3=InMemoryDataTree]
2016-07-28 09:30:21,340 [myid:] - INFO [main:SyncRequestProcessor@191] - Shutting down
2016-07-28 09:30:21,340 [myid:] - INFO [SyncThread:3:SyncRequestProcessor@169] - SyncRequestProcessor exited!
2016-07-28 09:30:21,340 [myid:] - INFO [ConnnectionExpirer:NIOServerCnxnFactory$ConnectionExpirerThread@583] - ConnnectionExpirerThread interrupted
2016-07-28 09:30:21,341 [myid:] - INFO [NIOServerCxnFactory.SelectorThread-0:NIOServerCnxnFactory$SelectorThread@420] - selector thread exitted run method
2016-07-28 09:30:21,341 [myid:] - INFO [NIOServerCxnFactory.AcceptThread:/127.0.0.1:11377:NIOServerCnxnFactory$AcceptThread@219] - accept thread exitted run method
2016-07-28 09:30:21,342 [myid:] - INFO [NIOServerCxnFactory.SelectorThread-1:NIOServerCnxnFactory$SelectorThread@420] - selector thread exitted run method
2016-07-28 09:30:21,342 [myid:] - INFO [/127.0.0.1:11379:QuorumCnxManager$Listener@661] - Leaving listener
2016-07-28 09:30:21,343 [myid:] - INFO [main:QuorumUtil@254] - Shutting down leader election QuorumPeer[myid=3](plain=/127.0.0.1:11377)(secure=disabled)
2016-07-28 09:30:21,343 [myid:] - INFO [main:QuorumUtil@259] - Waiting for QuorumPeer[myid=3](plain=/127.0.0.1:11377)(secure=disabled) to exit thread
2016-07-28 09:30:21,421 [myid:127.0.0.1:11273] - INFO [main-SendThread(127.0.0.1:11273):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:11273. Will not attempt to authenticate using SASL (unknown error)
2016-07-28 09:30:21,422 [myid:127.0.0.1:11273] - ERROR [main-SendThread(127.0.0.1:11273):ClientCnxnSocketNIO@287] - Unable to open socket to 127.0.0.1/127.0.0.1:11273
2016-07-28 09:30:21,422 [myid:127.0.0.1:11273] - WARN [main-SendThread(127.0.0.1:11273):ClientCnxn$SendThread@1235] - Session 0x222add090c60000 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.Net.connect0(Native Method)
at sun.nio.ch.Net.connect(Net.java:465)
at sun.nio.ch.Net.connect(Net.java:457)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670)
at org.apache.zookeeper.ClientCnxnSocketNIO.registerAndConnect(ClientCnxnSocketNIO.java:275)
at org.apache.zookeeper.ClientCnxnSocketNIO.connect(ClientCnxnSocketNIO.java:285)
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1098)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1133)
2016-07-28 09:30:21,451 [myid:127.0.0.1:11257] - INFO [main-SendThread(127.0.0.1:11257):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:11257. Will not attempt to authenticate using SASL (unknown error)
2016-07-28 09:30:21,451 [myid:127.0.0.1:11257] - ERROR [main-SendThread(127.0.0.1:11257):ClientCnxnSocketNIO@287] - Unable to open socket to 127.0.0.1/127.0.0.1:11257
2016-07-28 09:30:21,451 [myid:127.0.0.1:11257] - WARN [main-SendThread(127.0.0.1:11257):ClientCnxn$SendThread@1235] - Session 0x122adcf0ae70000 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.Net.connect0(Native Method)
at sun.nio.ch.Net.connect(Net.java:465)
at sun.nio.ch.Net.connect(Net.java:457)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670)
at org.apache.zookeeper.ClientCnxnSocketNIO.registerAndConnect(ClientCnxnSocketNIO.java:275)
at org.apache.zookeeper.ClientCnxnSocketNIO.connect(ClientCnxnSocketNIO.java:285)
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1098)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1133)
2016-07-28 09:30:21,671 [myid:127.0.0.1:11276] - INFO [main-SendThread(127.0.0.1:11276):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:11276. Will not attempt to authenticate using SASL (unknown error)
2016-07-28 09:30:21,671 [myid:127.0.0.1:11276] - ERROR [main-SendThread(127.0.0.1:11276):ClientCnxnSocketNIO@287] - Unable to open socket to 127.0.0.1/127.0.0.1:11276
2016-07-28 09:30:21,671 [myid:127.0.0.1:11276] - WARN [main-SendThread(127.0.0.1:11276):ClientCnxn$SendThread@1235] - Session 0x322add08d0d0000 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.Net.connect0(Native Method)
at sun.nio.ch.Net.connect(Net.java:465)
at sun.nio.ch.Net.connect(Net.java:457)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670)
at org.apache.zookeeper.ClientCnxnSocketNIO.registerAndConnect(ClientCnxnSocketNIO.java:275)
at org.apache.zookeeper.ClientCnxnSocketNIO.connect(ClientCnxnSocketNIO.java:285)
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1098)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1133)
2016-07-28 09:30:21,991 [myid:127.0.0.1:11306] - INFO [main-SendThread(127.0.0.1:11306):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:11306. Will not attempt to authenticate using SASL (unknown error)
2016-07-28 09:30:21,991 [myid:127.0.0.1:11306] - ERROR [main-SendThread(127.0.0.1:11306):ClientCnxnSocketNIO@287] - Unable to open socket to 127.0.0.1/127.0.0.1:11306
2016-07-28 09:30:21,991 [myid:127.0.0.1:11306] - WARN [main-SendThread(127.0.0.1:11306):ClientCnxn$SendThread@1235] - Session 0x222add109d00000 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.Net.connect0(Native Method)
at sun.nio.ch.Net.connect(Net.java:465)
at sun.nio.ch.Net.connect(Net.java:457)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670)
at org.apache.zookeeper.ClientCnxnSocketNIO.registerAndConnect(ClientCnxnSocketNIO.java:275)
at org.apache.zookeeper.ClientCnxnSocketNIO.connect(ClientCnxnSocketNIO.java:285)
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1098)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1133)
2016-07-28 09:30:22,091 [myid:127.0.0.1:11263] - INFO [main-SendThread(127.0.0.1:11263):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:11263. Will not attempt to authenticate using SASL (unknown error)
2016-07-28 09:30:22,091 [myid:127.0.0.1:11263] - ERROR [main-SendThread(127.0.0.1:11263):ClientCnxnSocketNIO@287] - Unable to open socket to 127.0.0.1/127.0.0.1:11263
2016-07-28 09:30:22,091 [myid:127.0.0.1:11263] - WARN [main-SendThread(127.0.0.1:11263):ClientCnxn$SendThread@1235] - Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.Net.connect0(Native Method)
at sun.nio.ch.Net.connect(Net.java:465)
at sun.nio.ch.Net.connect(Net.java:457)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670)
at org.apache.zookeeper.ClientCnxnSocketNIO.registerAndConnect(ClientCnxnSocketNIO.java:275)
at org.apache.zookeeper.ClientCnxnSocketNIO.connect(ClientCnxnSocketNIO.java:285)
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1098)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1133)
2016-07-28 09:30:22,111 [myid:127.0.0.1:11303] - INFO [main-SendThread(127.0.0.1:11303):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:11303. Will not attempt to authenticate using SASL (unknown error)
2016-07-28 09:30:22,111 [myid:127.0.0.1:11303] - ERROR [main-SendThread(127.0.0.1:11303):ClientCnxnSocketNIO@287] - Unable to open socket to 127.0.0.1/127.0.0.1:11303
2016-07-28 09:30:22,111 [myid:127.0.0.1:11303] - WARN [main-SendThread(127.0.0.1:11303):ClientCnxn$SendThread@1235] - Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.Net.connect0(Native Method)
at sun.nio.ch.Net.connect(Net.java:465)
at sun.nio.ch.Net.connect(Net.java:457)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670)
at org.apache.zookeeper.ClientCnxnSocketNIO.registerAndConnect(ClientCnxnSocketNIO.java:275)
at org.apache.zookeeper.ClientCnxnSocketNIO.connect(ClientCnxnSocketNIO.java:285)
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1098)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1133)
2016-07-28 09:30:22,241 [myid:127.0.0.1:11270] - INFO [main-SendThread(127.0.0.1:11270):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:11270. Will not attempt to authenticate using SASL (unknown error)
2016-07-28 09:30:22,241 [myid:127.0.0.1:11270] - ERROR [main-SendThread(127.0.0.1:11270):ClientCnxnSocketNIO@287] - Unable to open socket to 127.0.0.1/127.0.0.1:11270
2016-07-28 09:30:22,241 [myid:127.0.0.1:11270] - WARN [main-SendThread(127.0.0.1:11270):ClientCnxn$SendThread@1235] - Session 0x122add08ccf0000 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.Net.connect0(Native Method)
at sun.nio.ch.Net.connect(Net.java:465)
at sun.nio.ch.Net.connect(Net.java:457)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670)
at org.apache.zookeeper.ClientCnxnSocketNIO.registerAndConnect(ClientCnxnSocketNIO.java:275)
at org.apache.zookeeper.ClientCnxnSocketNIO.connect(ClientCnxnSocketNIO.java:285)
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1098)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1133)
2016-07-28 09:30:22,341 [myid:] - INFO [QuorumPeer[myid=3](plain=/127.0.0.1:11377)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id3,name1=replica.3,name2=Follower]
2016-07-28 09:30:22,341 [myid:] - INFO [QuorumPeer[myid=3](plain=/127.0.0.1:11377)(secure=disabled):Follower@198] - shutdown called
java.lang.Exception: shutdown Follower
at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:198)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1115)
2016-07-28 09:30:22,341 [myid:] - WARN [QuorumPeer[myid=3](plain=/127.0.0.1:11377)(secure=disabled):QuorumPeer@1158] - PeerState set to LOOKING
2016-07-28 09:30:22,341 [myid:] - WARN [QuorumPeer[myid=3](plain=/127.0.0.1:11377)(secure=disabled):QuorumPeer@1140] - QuorumPeer main thread exited
2016-07-28 09:30:22,341 [myid:] - INFO [QuorumPeer[myid=3](plain=/127.0.0.1:11377)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id3]
2016-07-28 09:30:22,342 [myid:] - INFO [QuorumPeer[myid=3](plain=/127.0.0.1:11377)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id3,name1=replica.3]
2016-07-28 09:30:22,342 [myid:] - INFO [QuorumPeer[myid=3](plain=/127.0.0.1:11377)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id3,name1=replica.1]
2016-07-28 09:30:22,342 [myid:] - INFO [QuorumPeer[myid=3](plain=/127.0.0.1:11377)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id3,name1=replica.2]
2016-07-28 09:30:22,342 [myid:] - INFO [QuorumPeer[myid=3](plain=/127.0.0.1:11377)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id3,name1=replica.5]
2016-07-28 09:30:22,342 [myid:] - INFO [main:QuorumUtil@250] - Shutting down quorum peer QuorumPeer[myid=4](plain=/127.0.0.1:11380)(secure=disabled)
2016-07-28 09:30:22,342 [myid:] - INFO [main:Follower@198] - shutdown called
java.lang.Exception: shutdown Follower
at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:198)
at org.apache.zookeeper.server.quorum.QuorumPeer.shutdown(QuorumPeer.java:1184)
at org.apache.zookeeper.test.QuorumUtil.shutdown(QuorumUtil.java:251)
at org.apache.zookeeper.test.QuorumUtil.shutdownAll(QuorumUtil.java:238)
at org.apache.zookeeper.test.QuorumUtil.tearDown(QuorumUtil.java:306)
at org.apache.zookeeper.test.ReconfigTest.tearDown(ReconfigTest.java:64)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33)
at org.junit.rules.TestWatchman$1.evaluate(TestWatchman.java:53)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:38)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:518)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1052)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:906)
2016-07-28 09:30:22,342 [myid:] - INFO [main:LearnerZooKeeperServer@165] - Shutting down
2016-07-28 09:30:22,342 [myid:] - INFO [main:ZooKeeperServer@498] - shutting down
2016-07-28 09:30:22,343 [myid:] - INFO [main:FollowerRequestProcessor@138] - Shutting down
2016-07-28 09:30:22,343 [myid:] - INFO [main:CommitProcessor@414] - Shutting down
2016-07-28 09:30:22,343 [myid:] - INFO [FollowerRequestProcessor:4:FollowerRequestProcessor@109] - FollowerRequestProcessor exited loop!
2016-07-28 09:30:22,343 [myid:] - INFO [CommitProcessor:4:CommitProcessor@299] - CommitProcessor exited loop!
2016-07-28 09:30:22,343 [myid:] - INFO [main:FinalRequestProcessor@479] - shutdown of request processor complete
2016-07-28 09:30:22,344 [myid:] - INFO [main:MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id4,name1=replica.4,name2=Follower,name3=InMemoryDataTree]
2016-07-28 09:30:22,344 [myid:] - INFO [main:SyncRequestProcessor@191] - Shutting down
2016-07-28 09:30:22,344 [myid:] - INFO [SyncThread:4:SyncRequestProcessor@169] - SyncRequestProcessor exited!
2016-07-28 09:30:22,344 [myid:] - INFO [ConnnectionExpirer:NIOServerCnxnFactory$ConnectionExpirerThread@583] - ConnnectionExpirerThread interrupted
2016-07-28 09:30:22,345 [myid:] - INFO [NIOServerCxnFactory.AcceptThread:/127.0.0.1:11380:NIOServerCnxnFactory$AcceptThread@219] - accept thread exitted run method
2016-07-28 09:30:22,345 [myid:] - INFO [NIOServerCxnFactory.SelectorThread-0:NIOServerCnxnFactory$SelectorThread@420] - selector thread exitted run method
2016-07-28 09:30:22,345 [myid:] - INFO [NIOServerCxnFactory.SelectorThread-1:NIOServerCnxnFactory$SelectorThread@420] - selector thread exitted run method
2016-07-28 09:30:22,346 [myid:] - INFO [/127.0.0.1:11382:QuorumCnxManager$Listener@661] - Leaving listener
2016-07-28 09:30:22,346 [myid:] - INFO [main:QuorumUtil@254] - Shutting down leader election QuorumPeer[myid=4](plain=/127.0.0.1:11380)(secure=disabled)
2016-07-28 09:30:22,346 [myid:] - INFO [main:QuorumUtil@259] - Waiting for QuorumPeer[myid=4](plain=/127.0.0.1:11380)(secure=disabled) to exit thread
2016-07-28 09:30:22,521 [myid:127.0.0.1:11309] - INFO [main-SendThread(127.0.0.1:11309):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:11309. Will not attempt to authenticate using SASL (unknown error)
2016-07-28 09:30:22,521 [myid:127.0.0.1:11309] - ERROR [main-SendThread(127.0.0.1:11309):ClientCnxnSocketNIO@287] - Unable to open socket to 127.0.0.1/127.0.0.1:11309
2016-07-28 09:30:22,521 [myid:127.0.0.1:11309] - WARN [main-SendThread(127.0.0.1:11309):ClientCnxn$SendThread@1235] - Session 0x322add109d50000 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.Net.connect0(Native Method)
at sun.nio.ch.Net.connect(Net.java:465)
at sun.nio.ch.Net.connect(Net.java:457)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670)
at org.apache.zookeeper.ClientCnxnSocketNIO.registerAndConnect(ClientCnxnSocketNIO.java:275)
at org.apache.zookeeper.ClientCnxnSocketNIO.connect(ClientCnxnSocketNIO.java:285)
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1098)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1133)
2016-07-28 09:30:22,631 [myid:127.0.0.1:11273] - INFO [main-SendThread(127.0.0.1:11273):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:11273. Will not attempt to authenticate using SASL (unknown error)
2016-07-28 09:30:22,631 [myid:127.0.0.1:11273] - ERROR [main-SendThread(127.0.0.1:11273):ClientCnxnSocketNIO@287] - Unable to open socket to 127.0.0.1/127.0.0.1:11273
2016-07-28 09:30:22,631 [myid:127.0.0.1:11273] - WARN [main-SendThread(127.0.0.1:11273):ClientCnxn$SendThread@1235] - Session 0x222add090c60000 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.Net.connect0(Native Method)
at sun.nio.ch.Net.connect(Net.java:465)
at sun.nio.ch.Net.connect(Net.java:457)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670)
at org.apache.zookeeper.ClientCnxnSocketNIO.registerAndConnect(ClientCnxnSocketNIO.java:275)
at org.apache.zookeeper.ClientCnxnSocketNIO.connect(ClientCnxnSocketNIO.java:285)
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1098)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1133)
2016-07-28 09:30:22,761 [myid:] - INFO [WorkerReceiver[myid=4]:FastLeaderElection$Messenger$WorkerReceiver@440] - WorkerReceiver is down
2016-07-28 09:30:22,761 [myid:] - INFO [WorkerSender[myid=4]:FastLeaderElection$Messenger$WorkerSender@470] - WorkerSender is down
2016-07-28 09:30:22,771 [myid:] - INFO [WorkerReceiver[myid=1]:FastLeaderElection$Messenger$WorkerReceiver@440] - WorkerReceiver is down
2016-07-28 09:30:22,771 [myid:] - INFO [WorkerSender[myid=1]:FastLeaderElection$Messenger$WorkerSender@470] - WorkerSender is down
2016-07-28 09:30:22,771 [myid:] - INFO [WorkerSender[myid=2]:FastLeaderElection$Messenger$WorkerSender@470] - WorkerSender is down
2016-07-28 09:30:22,771 [myid:] - INFO [WorkerReceiver[myid=3]:FastLeaderElection$Messenger$WorkerReceiver@440] - WorkerReceiver is down
2016-07-28 09:30:22,771 [myid:] - INFO [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@440] - WorkerReceiver is down
2016-07-28 09:30:22,771 [myid:] - INFO [WorkerSender[myid=3]:FastLeaderElection$Messenger$WorkerSender@470] - WorkerSender is down
2016-07-28 09:30:23,061 [myid:127.0.0.1:11260] - INFO [main-SendThread(127.0.0.1:11260):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:11260. Will not attempt to authenticate using SASL (unknown error)
2016-07-28 09:30:23,061 [myid:127.0.0.1:11260] - ERROR [main-SendThread(127.0.0.1:11260):ClientCnxnSocketNIO@287] - Unable to open socket to 127.0.0.1/127.0.0.1:11260
2016-07-28 09:30:23,061 [myid:127.0.0.1:11260] - WARN [main-SendThread(127.0.0.1:11260):ClientCnxn$SendThread@1235] - Session 0x222adcf0da20000 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.Net.connect0(Native Method)
at sun.nio.ch.Net.connect(Net.java:465)
at sun.nio.ch.Net.connect(Net.java:457)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670)
at org.apache.zookeeper.ClientCnxnSocketNIO.registerAndConnect(ClientCnxnSocketNIO.java:275)
at org.apache.zookeeper.ClientCnxnSocketNIO.connect(ClientCnxnSocketNIO.java:285)
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1098)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1133)
2016-07-28 09:30:23,211 [myid:127.0.0.1:11263] - INFO [main-SendThread(127.0.0.1:11263):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:11263. Will not attempt to authenticate using SASL (unknown error)
2016-07-28 09:30:23,211 [myid:127.0.0.1:11263] - ERROR [main-SendThread(127.0.0.1:11263):ClientCnxnSocketNIO@287] - Unable to open socket to 127.0.0.1/127.0.0.1:11263
2016-07-28 09:30:23,211 [myid:127.0.0.1:11263] - WARN [main-SendThread(127.0.0.1:11263):ClientCnxn$SendThread@1235] - Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.Net.connect0(Native Method)
at sun.nio.ch.Net.connect(Net.java:465)
at sun.nio.ch.Net.connect(Net.java:457)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670)
at org.apache.zookeeper.ClientCnxnSocketNIO.registerAndConnect(ClientCnxnSocketNIO.java:275)
at org.apache.zookeeper.ClientCnxnSocketNIO.connect(ClientCnxnSocketNIO.java:285)
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1098)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1133)
2016-07-28 09:30:23,231 [myid:127.0.0.1:11303] - INFO [main-SendThread(127.0.0.1:11303):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:11303. Will not attempt to authenticate using SASL (unknown error)
2016-07-28 09:30:23,231 [myid:127.0.0.1:11276] - INFO [main-SendThread(127.0.0.1:11276):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:11276. Will not attempt to authenticate using SASL (unknown error)
2016-07-28 09:30:23,231 [myid:127.0.0.1:11303] - ERROR [main-SendThread(127.0.0.1:11303):ClientCnxnSocketNIO@287] - Unable to open socket to 127.0.0.1/127.0.0.1:11303
2016-07-28 09:30:23,231 [myid:127.0.0.1:11303] - WARN [main-SendThread(127.0.0.1:11303):ClientCnxn$SendThread@1235] - Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.Net.connect0(Native Method)
at sun.nio.ch.Net.connect(Net.java:465)
at sun.nio.ch.Net.connect(Net.java:457)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670)
at org.apache.zookeeper.ClientCnxnSocketNIO.registerAndConnect(ClientCnxnSocketNIO.java:275)
at org.apache.zookeeper.ClientCnxnSocketNIO.connect(ClientCnxnSocketNIO.java:285)
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1098)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1133)
2016-07-28 09:30:23,231 [myid:127.0.0.1:11276] - ERROR [main-SendThread(127.0.0.1:11276):ClientCnxnSocketNIO@287] - Unable to open socket to 127.0.0.1/127.0.0.1:11276
2016-07-28 09:30:23,232 [myid:127.0.0.1:11276] - WARN [main-SendThread(127.0.0.1:11276):ClientCnxn$SendThread@1235] - Session 0x322add08d0d0000 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.Net.connect0(Native Method)
at sun.nio.ch.Net.connect(Net.java:465)
at sun.nio.ch.Net.connect(Net.java:457)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670)
at org.apache.zookeeper.ClientCnxnSocketNIO.registerAndConnect(ClientCnxnSocketNIO.java:275)
at org.apache.zookeeper.ClientCnxnSocketNIO.connect(ClientCnxnSocketNIO.java:285)
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1098)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1133)
2016-07-28 09:30:23,341 [myid:127.0.0.1:11257] - INFO [main-SendThread(127.0.0.1:11257):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:11257. Will not attempt to authenticate using SASL (unknown error)
2016-07-28 09:30:23,341 [myid:127.0.0.1:11257] - ERROR [main-SendThread(127.0.0.1:11257):ClientCnxnSocketNIO@287] - Unable to open socket to 127.0.0.1/127.0.0.1:11257
2016-07-28 09:30:23,341 [myid:127.0.0.1:11257] - WARN [main-SendThread(127.0.0.1:11257):ClientCnxn$SendThread@1235] - Session 0x122adcf0ae70000 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.Net.connect0(Native Method)
at sun.nio.ch.Net.connect(Net.java:465)
at sun.nio.ch.Net.connect(Net.java:457)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670)
at org.apache.zookeeper.ClientCnxnSocketNIO.registerAndConnect(ClientCnxnSocketNIO.java:275)
at org.apache.zookeeper.ClientCnxnSocketNIO.connect(ClientCnxnSocketNIO.java:285)
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1098)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1133)
2016-07-28 09:30:23,351 [myid:] - INFO [QuorumPeer[myid=4](plain=/127.0.0.1:11380)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id4,name1=replica.4,name2=Follower]
2016-07-28 09:30:23,351 [myid:] - INFO [QuorumPeer[myid=4](plain=/127.0.0.1:11380)(secure=disabled):Follower@198] - shutdown called
java.lang.Exception: shutdown Follower
at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:198)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1115)
2016-07-28 09:30:23,351 [myid:] - WARN [QuorumPeer[myid=4](plain=/127.0.0.1:11380)(secure=disabled):QuorumPeer@1158] - PeerState set to LOOKING
2016-07-28 09:30:23,351 [myid:] - WARN [QuorumPeer[myid=4](plain=/127.0.0.1:11380)(secure=disabled):QuorumPeer@1140] - QuorumPeer main thread exited
2016-07-28 09:30:23,351 [myid:] - INFO [QuorumPeer[myid=4](plain=/127.0.0.1:11380)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id4]
2016-07-28 09:30:23,352 [myid:] - INFO [QuorumPeer[myid=4](plain=/127.0.0.1:11380)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id4,name1=replica.4]
2016-07-28 09:30:23,352 [myid:] - INFO [QuorumPeer[myid=4](plain=/127.0.0.1:11380)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id4,name1=replica.1]
2016-07-28 09:30:23,352 [myid:] - INFO [QuorumPeer[myid=4](plain=/127.0.0.1:11380)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id4,name1=replica.2]
2016-07-28 09:30:23,352 [myid:] - INFO [QuorumPeer[myid=4](plain=/127.0.0.1:11380)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id4,name1=replica.3]
2016-07-28 09:30:23,352 [myid:] - INFO [QuorumPeer[myid=4](plain=/127.0.0.1:11380)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id4,name1=replica.5]
2016-07-28 09:30:23,352 [myid:] - INFO [main:QuorumUtil@250] - Shutting down quorum peer QuorumPeer[myid=5](plain=/127.0.0.1:11383)(secure=disabled)
2016-07-28 09:30:23,352 [myid:] - INFO [main:Leader@617] - Shutting down
2016-07-28 09:30:23,352 [myid:] - INFO [main:Leader@623] - Shutdown called
java.lang.Exception: shutdown Leader! reason: quorum Peer shutdown
at org.apache.zookeeper.server.quorum.Leader.shutdown(Leader.java:623)
at org.apache.zookeeper.server.quorum.QuorumPeer.shutdown(QuorumPeer.java:1181)
at org.apache.zookeeper.test.QuorumUtil.shutdown(QuorumUtil.java:251)
at org.apache.zookeeper.test.QuorumUtil.shutdownAll(QuorumUtil.java:238)
at org.apache.zookeeper.test.QuorumUtil.tearDown(QuorumUtil.java:306)
at org.apache.zookeeper.test.ReconfigTest.tearDown(ReconfigTest.java:64)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33)
at org.junit.rules.TestWatchman$1.evaluate(TestWatchman.java:53)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:38)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:518)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1052)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:906)
2016-07-28 09:30:23,353 [myid:] - INFO [main:ZooKeeperServer@498] - shutting down
2016-07-28 09:30:23,353 [myid:] - INFO [main:SessionTrackerImpl@232] - Shutting down
2016-07-28 09:30:23,353 [myid:] - INFO [main:LeaderRequestProcessor@77] - Shutting down
2016-07-28 09:30:23,353 [myid:] - INFO [main:PrepRequestProcessor@965] - Shutting down
2016-07-28 09:30:23,353 [myid:] - INFO [LearnerCnxAcceptor-/127.0.0.1:11384:Leader$LearnerCnxAcceptor@373] - exception while shutting down acceptor: java.net.SocketException: Socket closed
2016-07-28 09:30:23,353 [myid:] - INFO [ProcessThread(sid:5 cport:-1)::PrepRequestProcessor@154] - PrepRequestProcessor exited loop!
2016-07-28 09:30:23,353 [myid:] - INFO [main:ProposalRequestProcessor@88] - Shutting down
2016-07-28 09:30:23,354 [myid:] - INFO [main:CommitProcessor@414] - Shutting down
2016-07-28 09:30:23,354 [myid:] - INFO [CommitProcessor:5:CommitProcessor@299] - CommitProcessor exited loop!
2016-07-28 09:30:23,354 [myid:] - INFO [main:Leader$ToBeAppliedRequestProcessor@918] - Shutting down
2016-07-28 09:30:23,354 [myid:] - INFO [main:FinalRequestProcessor@479] - shutdown of request processor complete
2016-07-28 09:30:23,354 [myid:] - INFO [main:SyncRequestProcessor@191] - Shutting down
2016-07-28 09:30:23,354 [myid:] - INFO [SyncThread:5:SyncRequestProcessor@169] - SyncRequestProcessor exited!
2016-07-28 09:30:23,355 [myid:] - INFO [main:MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id5,name1=replica.5,name2=Leader,name3=InMemoryDataTree]
2016-07-28 09:30:23,355 [myid:] - WARN [LearnerHandler-/127.0.0.1:51992:LearnerHandler@619] - ******* GOODBYE /127.0.0.1:51992 ********
2016-07-28 09:30:23,355 [myid:] - WARN [LearnerHandler-/127.0.0.1:51977:LearnerHandler@619] - ******* GOODBYE /127.0.0.1:51977 ********
2016-07-28 09:30:23,355 [myid:] - WARN [LearnerHandler-/127.0.0.1:51992:LearnerHandler@903] - Ignoring unexpected exception
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1219)
at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:340)
at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:338)
at org.apache.zookeeper.server.quorum.LearnerHandler.shutdown(LearnerHandler.java:901)
at org.apache.zookeeper.server.quorum.LearnerHandler.run(LearnerHandler.java:622)
2016-07-28 09:30:23,355 [myid:] - WARN [LearnerHandler-/127.0.0.1:51976:LearnerHandler@619] - ******* GOODBYE /127.0.0.1:51976 ********
2016-07-28 09:30:23,355 [myid:] - WARN [LearnerHandler-/127.0.0.1:51975:LearnerHandler@619] - ******* GOODBYE /127.0.0.1:51975 ********
2016-07-28 09:30:23,356 [myid:] - INFO [ConnnectionExpirer:NIOServerCnxnFactory$ConnectionExpirerThread@583] - ConnnectionExpirerThread interrupted
2016-07-28 09:30:23,356 [myid:] - INFO [NIOServerCxnFactory.SelectorThread-0:NIOServerCnxnFactory$SelectorThread@420] - selector thread exitted run method
2016-07-28 09:30:23,355 [myid:] - WARN [LearnerHandler-/127.0.0.1:51977:LearnerHandler@903] - Ignoring unexpected exception
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1219)
at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:340)
at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:338)
at org.apache.zookeeper.server.quorum.LearnerHandler.shutdown(LearnerHandler.java:901)
at org.apache.zookeeper.server.quorum.LearnerHandler.run(LearnerHandler.java:622)
2016-07-28 09:30:23,356 [myid:] - INFO [NIOServerCxnFactory.AcceptThread:/127.0.0.1:11383:NIOServerCnxnFactory$AcceptThread@219] - accept thread exitted run method
2016-07-28 09:30:23,356 [myid:] - INFO [NIOServerCxnFactory.SelectorThread-1:NIOServerCnxnFactory$SelectorThread@420] - selector thread exitted run method
2016-07-28 09:30:23,356 [myid:] - WARN [LearnerHandler-/127.0.0.1:51975:LearnerHandler@903] - Ignoring unexpected exception
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1219)
at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:340)
at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:338)
at org.apache.zookeeper.server.quorum.LearnerHandler.shutdown(LearnerHandler.java:901)
at org.apache.zookeeper.server.quorum.LearnerHandler.run(LearnerHandler.java:622)
2016-07-28 09:30:23,356 [myid:] - WARN [LearnerHandler-/127.0.0.1:51976:LearnerHandler@903] - Ignoring unexpected exception
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1219)
at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:340)
at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:338)
at org.apache.zookeeper.server.quorum.LearnerHandler.shutdown(LearnerHandler.java:901)
at org.apache.zookeeper.server.quorum.LearnerHandler.run(LearnerHandler.java:622)
2016-07-28 09:30:23,357 [myid:] - INFO [/127.0.0.1:11385:QuorumCnxManager$Listener@661] - Leaving listener
2016-07-28 09:30:23,357 [myid:] - INFO [QuorumPeer[myid=5](plain=/127.0.0.1:11383)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id5,name1=replica.5,name2=Leader]
2016-07-28 09:30:23,358 [myid:] - INFO [main:QuorumUtil@254] - Shutting down leader election QuorumPeer[myid=5](plain=/127.0.0.1:11383)(secure=disabled)
2016-07-28 09:30:23,358 [myid:] - WARN [QuorumPeer[myid=5](plain=/127.0.0.1:11383)(secure=disabled):QuorumPeer@1127] - Unexpected exception
java.lang.InterruptedException
at java.lang.Object.wait(Native Method)
at org.apache.zookeeper.server.quorum.Leader.lead(Leader.java:561)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1124)
2016-07-28 09:30:23,358 [myid:] - INFO [QuorumPeer[myid=5](plain=/127.0.0.1:11383)(secure=disabled):Leader@617] - Shutting down
2016-07-28 09:30:23,358 [myid:] - WARN [QuorumPeer[myid=5](plain=/127.0.0.1:11383)(secure=disabled):QuorumPeer@1158] - PeerState set to LOOKING
2016-07-28 09:30:23,358 [myid:] - WARN [QuorumPeer[myid=5](plain=/127.0.0.1:11383)(secure=disabled):QuorumPeer@1140] - QuorumPeer main thread exited
2016-07-28 09:30:23,358 [myid:] - INFO [main:QuorumUtil@259] - Waiting for QuorumPeer[myid=5](plain=/127.0.0.1:11383)(secure=disabled) to exit thread
2016-07-28 09:30:23,358 [myid:] - INFO [QuorumPeer[myid=5](plain=/127.0.0.1:11383)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id5]
2016-07-28 09:30:23,359 [myid:] - INFO [QuorumPeer[myid=5](plain=/127.0.0.1:11383)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id5,name1=replica.5]
2016-07-28 09:30:23,359 [myid:] - INFO [QuorumPeer[myid=5](plain=/127.0.0.1:11383)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id5,name1=replica.1]
2016-07-28 09:30:23,359 [myid:] - INFO [QuorumPeer[myid=5](plain=/127.0.0.1:11383)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id5,name1=replica.2]
2016-07-28 09:30:23,359 [myid:] - INFO [QuorumPeer[myid=5](plain=/127.0.0.1:11383)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id5,name1=replica.3]
2016-07-28 09:30:23,359 [myid:] - INFO [main:FourLetterWordMain@85] - connecting to 127.0.0.1 11371
2016-07-28 09:30:23,359 [myid:] - INFO [main:QuorumUtil@243] - 127.0.0.1:11371 is no longer accepting client connections
2016-07-28 09:30:23,359 [myid:] - INFO [main:FourLetterWordMain@85] - connecting to 127.0.0.1 11374
2016-07-28 09:30:23,360 [myid:] - INFO [main:QuorumUtil@243] - 127.0.0.1:11374 is no longer accepting client connections
2016-07-28 09:30:23,360 [myid:] - INFO [main:FourLetterWordMain@85] - connecting to 127.0.0.1 11377
2016-07-28 09:30:23,360 [myid:] - INFO [main:QuorumUtil@243] - 127.0.0.1:11377 is no longer accepting client connections
2016-07-28 09:30:23,360 [myid:] - INFO [main:FourLetterWordMain@85] - connecting to 127.0.0.1 11380
2016-07-28 09:30:23,360 [myid:] - INFO [main:QuorumUtil@243] - 127.0.0.1:11380 is no longer accepting client connections
2016-07-28 09:30:23,360 [myid:] - INFO [main:FourLetterWordMain@85] - connecting to 127.0.0.1 11383
2016-07-28 09:30:23,360 [myid:] - INFO [main:QuorumUtil@243] - 127.0.0.1:11383 is no longer accepting client connections
2016-07-28 09:30:23,361 [myid:] - INFO [main:ZKTestCase$1@65] - SUCCEEDED testRemoveOneAsynchronous
2016-07-28 09:30:23,361 [myid:] - INFO [main:ZKTestCase$1@60] - FINISHED testRemoveOneAsynchronous
{noformat}

{noformat}
Failed

org.apache.zookeeper.test.ReconfigTest.testPortChangeToBlockedPortLeader

Failing for the past 5 builds (Since Failed#1241 )
Took 1 min 38 sec.
Error Message

client could not connect to reestablished quorum: giving up after 30+ seconds.
Stacktrace

junit.framework.AssertionFailedError: client could not connect to reestablished quorum: giving up after 30+ seconds.
at org.apache.zookeeper.test.ReconfigTest.testNormalOperation(ReconfigTest.java:173)
at org.apache.zookeeper.test.ReconfigTest.testPortChangeToBlockedPort(ReconfigTest.java:732)
at org.apache.zookeeper.test.ReconfigTest.testPortChangeToBlockedPortLeader(ReconfigTest.java:662)
at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:79)
Standard Output

2016-07-28 09:24:37,121 [myid:] - INFO [main:JUnit4ZKTestRunner@47] - No test.method specified. using default methods.
2016-07-28 09:24:37,185 [myid:] - INFO [main:JUnit4ZKTestRunner@47] - No test.method specified. using default methods.
2016-07-28 09:24:37,200 [myid:] - INFO [main:ZKTestCase$1@55] - STARTING testQuorumSystemChange
2016-07-28 09:24:37,202 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@77] - RUNNING TEST METHOD testQuorumSystemChange
2016-07-28 09:24:37,577 [myid:] - INFO [main:PortAssignment@157] - Single test process using ports from 11221 - 32767.
2016-07-28 09:24:37,577 [myid:] - INFO [main:PortAssignment@85] - Assigned port 11222 from range 11221 - 32767.
2016-07-28 09:24:37,579 [myid:] - INFO [main:PortAssignment@85] - Assigned port 11223 from range 11221 - 32767.
2016-07-28 09:24:37,579 [myid:] - INFO [main:PortAssignment@85] - Assigned port 11224 from range 11221 - 32767.
2016-07-28 09:24:37,580 [myid:] - INFO [main:PortAssignment@85] - Assigned port 11225 from range 11221 - 32767.
2016-07-28 09:24:37,581 [myid:] - INFO [main:PortAssignment@85] - Assigned port 11226 from range 11221 - 32767.
2016-07-28 09:24:37,581 [myid:] - INFO [main:PortAssignment@85] - Assigned port 11227 from range 11221 - 32767.
2016-07-28 09:24:37,581 [myid:] - INFO [main:PortAssignment@85] - Assigned port 11228 from range 11221 - 32767.
2016-07-28 09:24:37,582 [myid:] - INFO [main:PortAssignment@85] - Assigned port 11229 from range 11221 - 32767.
2016-07-28 09:24:37,582 [myid:] - INFO [main:PortAssignment@85] - Assigned port 11230 from range 11221 - 32767.
2016-07-28 09:24:37,582 [myid:] - INFO [main:PortAssignment@85] - Assigned port 11231 from range 11221 - 32767.
2016-07-28 09:24:37,582 [myid:] - INFO [main:PortAssignment@85] - Assigned port 11232 from range 11221 - 32767.
2016-07-28 09:24:37,584 [myid:] - INFO [main:PortAssignment@85] - Assigned port 11233 from range 11221 - 32767.
2016-07-28 09:24:37,585 [myid:] - INFO [main:PortAssignment@85] - Assigned port 11234 from range 11221 - 32767.
2016-07-28 09:24:37,585 [myid:] - INFO [main:PortAssignment@85] - Assigned port 11235 from range 11221 - 32767.
2016-07-28 09:24:37,585 [myid:] - INFO [main:PortAssignment@85] - Assigned port 11236 from range 11221 - 32767.
2016-07-28 09:24:37,585 [myid:] - INFO [main:PortAssignment@85] - Assigned port 11237 from range 11221 - 32767.
2016-07-28 09:24:37,586 [myid:] - INFO [main:PortAssignment@85] - Assigned port 11238 from range 11221 - 32767.
2016-07-28 09:24:37,586 [myid:] - INFO [main:PortAssignment@85] - Assigned port 11239 from range 11221 - 32767.
2016-07-28 09:24:37,586 [myid:] - INFO [main:PortAssignment@85] - Assigned port 11240 from range 11221 - 32767.
2016-07-28 09:24:37,587 [myid:] - INFO [main:PortAssignment@85] - Assigned port 11241 from range 11221 - 32767.
2016-07-28 09:24:37,587 [myid:] - INFO [main:PortAssignment@85] - Assigned port 11242 from range 11221 - 32767.
2016-07-28 09:24:37,587 [myid:] - INFO [main:QuorumUtil@116] - Creating QuorumPeer 1; public port 11222
2016-07-28 09:24:37,609 [myid:] - INFO [main:NIOServerCnxnFactory@673] - Configuring NIO connection handler with 10s sessionless connection timeout, 2 selector thread(s), 16 worker threads, and 64 kB direct buffers.
2016-07-28 09:24:37,615 [myid:] - INFO [main:NIOServerCnxnFactory@686] - binding to port /127.0.0.1:11222
2016-07-28 09:24:37,638 [myid:] - INFO [main:QuorumUtil@116] - Creating QuorumPeer 2; public port 11225
2016-07-28 09:24:37,638 [myid:] - INFO [main:NIOServerCnxnFactory@673] - Configuring NIO connection handler with 10s sessionless connection timeout, 2 selector thread(s), 16 worker threads, and 64 kB direct buffers.
2016-07-28 09:24:37,639 [myid:] - INFO [main:NIOServerCnxnFactory@686] - binding to port /127.0.0.1:11225
2016-07-28 09:24:37,639 [myid:] - INFO [main:QuorumUtil@116] - Creating QuorumPeer 3; public port 11228
2016-07-28 09:24:37,640 [myid:] - INFO [main:NIOServerCnxnFactory@673] - Configuring NIO connection handler with 10s sessionless connection timeout, 2 selector thread(s), 16 worker threads, and 64 kB direct buffers.
2016-07-28 09:24:37,640 [myid:] - INFO [main:NIOServerCnxnFactory@686] - binding to port /127.0.0.1:11228
2016-07-28 09:24:37,641 [myid:] - INFO [main:QuorumUtil@116] - Creating QuorumPeer 4; public port 11231
2016-07-28 09:24:37,641 [myid:] - INFO [main:NIOServerCnxnFactory@673] - Configuring NIO connection handler with 10s sessionless connection timeout, 2 selector thread(s), 16 worker threads, and 64 kB direct buffers.
2016-07-28 09:24:37,641 [myid:] - INFO [main:NIOServerCnxnFactory@686] - binding to port /127.0.0.1:11231
2016-07-28 09:24:37,642 [myid:] - INFO [main:QuorumUtil@116] - Creating QuorumPeer 5; public port 11234
2016-07-28 09:24:37,642 [myid:] - INFO [main:NIOServerCnxnFactory@673] - Configuring NIO connection handler with 10s sessionless connection timeout, 2 selector thread(s), 16 worker threads, and 64 kB direct buffers.
2016-07-28 09:24:37,642 [myid:] - INFO [main:NIOServerCnxnFactory@686] - binding to port /127.0.0.1:11234
2016-07-28 09:24:37,643 [myid:] - INFO [main:QuorumUtil@116] - Creating QuorumPeer 6; public port 11237
2016-07-28 09:24:37,643 [myid:] - INFO [main:NIOServerCnxnFactory@673] - Configuring NIO connection handler with 10s sessionless connection timeout, 2 selector thread(s), 16 worker threads, and 64 kB direct buffers.
2016-07-28 09:24:37,644 [myid:] - INFO [main:NIOServerCnxnFactory@686] - binding to port /127.0.0.1:11237
2016-07-28 09:24:37,644 [myid:] - INFO [main:QuorumUtil@116] - Creating QuorumPeer 7; public port 11240
2016-07-28 09:24:37,644 [myid:] - INFO [main:NIOServerCnxnFactory@673] - Configuring NIO connection handler with 10s sessionless connection timeout, 2 selector thread(s), 16 worker threads, and 64 kB direct buffers.
2016-07-28 09:24:37,645 [myid:] - INFO [main:NIOServerCnxnFactory@686] - binding to port /127.0.0.1:11240
2016-07-28 09:24:37,646 [myid:] - INFO [main:QuorumUtil@250] - Shutting down quorum peer QuorumPeer
2016-07-28 09:24:37,646 [myid:] - INFO [main:QuorumUtil@257] - No election available to shutdown QuorumPeer
2016-07-28 09:24:37,646 [myid:] - INFO [main:QuorumUtil@259] - Waiting for QuorumPeer to exit thread
2016-07-28 09:24:37,646 [myid:] - INFO [main:QuorumUtil@250] - Shutting down quorum peer QuorumPeer
2016-07-28 09:24:37,647 [myid:] - INFO [main:QuorumUtil@257] - No election available to shutdown QuorumPeer
2016-07-28 09:24:37,647 [myid:] - INFO [main:QuorumUtil@259] - Waiting for QuorumPeer to exit thread
2016-07-28 09:24:37,647 [myid:] - INFO [main:QuorumUtil@250] - Shutting down quorum peer QuorumPeer
2016-07-28 09:24:37,647 [myid:] - INFO [main:QuorumUtil@257] - No election available to shutdown QuorumPeer
2016-07-28 09:24:37,647 [myid:] - INFO [main:QuorumUtil@259] - Waiting for QuorumPeer to exit thread
2016-07-28 09:24:37,647 [myid:] - INFO [main:QuorumUtil@250] - Shutting down quorum peer QuorumPeer
2016-07-28 09:24:37,647 [myid:] - INFO [main:QuorumUtil@257] - No election available to shutdown QuorumPeer
2016-07-28 09:24:37,647 [myid:] - INFO [main:QuorumUtil@259] - Waiting for QuorumPeer to exit thread
2016-07-28 09:24:37,648 [myid:] - INFO [main:QuorumUtil@250] - Shutting down quorum peer QuorumPeer
2016-07-28 09:24:37,648 [myid:] - INFO [main:QuorumUtil@257] - No election available to shutdown QuorumPeer
2016-07-28 09:24:37,648 [myid:] - INFO [main:QuorumUtil@259] - Waiting for QuorumPeer to exit thread
2016-07-28 09:24:37,648 [myid:] - INFO [main:QuorumUtil@250] - Shutting down quorum peer QuorumPeer
2016-07-28 09:24:37,648 [myid:] - INFO [main:QuorumUtil@257] - No election available to shutdown QuorumPeer
2016-07-28 09:24:37,648 [myid:] - INFO [main:QuorumUtil@259] - Waiting for QuorumPeer to exit thread
2016-07-28 09:24:37,648 [myid:] - INFO [main:QuorumUtil@250] - Shutting down quorum peer QuorumPeer
2016-07-28 09:24:37,649 [myid:] - INFO [main:QuorumUtil@257] - No election available to shutdown QuorumPeer
2016-07-28 09:24:37,649 [myid:] - INFO [main:QuorumUtil@259] - Waiting for QuorumPeer to exit thread
2016-07-28 09:24:37,651 [myid:] - INFO [main:FourLetterWordMain@85] - connecting to 127.0.0.1 11222
2016-07-28 09:24:37,651 [myid:] - INFO [main:QuorumUtil@243] - 127.0.0.1:11222 is no longer accepting client connections
2016-07-28 09:24:37,651 [myid:] - INFO [main:FourLetterWordMain@85] - connecting to 127.0.0.1 11225
2016-07-28 09:24:37,652 [myid:] - INFO [main:QuorumUtil@243] - 127.0.0.1:11225 is no longer accepting client connections
2016-07-28 09:24:37,652 [myid:] - INFO [main:FourLetterWordMain@85] - connecting to 127.0.0.1 11228
2016-07-28 09:24:37,652 [myid:] - INFO [main:QuorumUtil@243] - 127.0.0.1:11228 is no longer accepting client connections
2016-07-28 09:24:37,652 [myid:] - INFO [main:FourLetterWordMain@85] - connecting to 127.0.0.1 11231
2016-07-28 09:24:37,653 [myid:] - INFO [main:QuorumUtil@243] - 127.0.0.1:11231 is no longer accepting client connections
2016-07-28 09:24:37,653 [myid:] - INFO [main:FourLetterWordMain@85] - connecting to 127.0.0.1 11234
2016-07-28 09:24:37,653 [myid:] - INFO [main:QuorumUtil@243] - 127.0.0.1:11234 is no longer accepting client connections
2016-07-28 09:24:37,653 [myid:] - INFO [main:FourLetterWordMain@85] - connecting to 127.0.0.1 11237
2016-07-28 09:24:37,653 [myid:] - INFO [main:QuorumUtil@243] - 127.0.0.1:11237 is no longer accepting client connections
2016-07-28 09:24:37,654 [myid:] - INFO [main:FourLetterWordMain@85] - connecting to 127.0.0.1 11240
2016-07-28 09:24:37,654 [myid:] - INFO [main:QuorumUtil@243] - 127.0.0.1:11240 is no longer accepting client connections
2016-07-28 09:24:37,654 [myid:] - INFO [main:QuorumUtil@203] - Creating QuorumPeer 1; public port 11222
2016-07-28 09:24:37,654 [myid:] - INFO [main:NIOServerCnxnFactory@673] - Configuring NIO connection handler with 10s sessionless connection timeout, 2 selector thread(s), 16 worker threads, and 64 kB direct buffers.
2016-07-28 09:24:37,655 [myid:] - INFO [main:NIOServerCnxnFactory@686] - binding to port /127.0.0.1:11222
2016-07-28 09:24:37,660 [myid:] - INFO [main:QuorumPeer@776] - currentEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
2016-07-28 09:24:37,708 [myid:] - INFO [main:QuorumPeer@791] - acceptedEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
2016-07-28 09:24:37,754 [myid:] - INFO [main:QuorumUtil@146] - Started QuorumPeer 1
2016-07-28 09:24:37,754 [myid:] - INFO [main:QuorumUtil@203] - Creating QuorumPeer 2; public port 11225
2016-07-28 09:24:37,755 [myid:] - INFO [main:NIOServerCnxnFactory@673] - Configuring NIO connection handler with 10s sessionless connection timeout, 2 selector thread(s), 16 worker threads, and 64 kB direct buffers.
2016-07-28 09:24:37,755 [myid:] - INFO [main:NIOServerCnxnFactory@686] - binding to port /127.0.0.1:11225
2016-07-28 09:24:37,756 [myid:] - INFO [main:QuorumPeer@776] - currentEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
2016-07-28 09:24:37,759 [myid:] - INFO [QuorumPeerListener:QuorumCnxManager$Listener@632] - My election bind port: /127.0.0.1:11224
2016-07-28 09:24:37,771 [myid:] - INFO [main:QuorumPeer@791] - acceptedEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
2016-07-28 09:24:37,780 [myid:] - INFO [main:QuorumUtil@146] - Started QuorumPeer 2
2016-07-28 09:24:37,780 [myid:] - INFO [main:QuorumUtil@203] - Creating QuorumPeer 3; public port 11228
2016-07-28 09:24:37,781 [myid:] - INFO [main:NIOServerCnxnFactory@673] - Configuring NIO connection handler with 10s sessionless connection timeout, 2 selector thread(s), 16 worker threads, and 64 kB direct buffers.
2016-07-28 09:24:37,781 [myid:] - INFO [main:NIOServerCnxnFactory@686] - binding to port /127.0.0.1:11228
2016-07-28 09:24:37,781 [myid:] - INFO [QuorumPeerListener:QuorumCnxManager$Listener@632] - My election bind port: /127.0.0.1:11227
2016-07-28 09:24:37,888 [myid:] - INFO [main:QuorumPeer@776] - currentEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
2016-07-28 09:24:37,894 [myid:] - INFO [QuorumPeer[myid=1](plain=/127.0.0.1:11222)(secure=disabled):QuorumPeer@1033] - LOOKING
2016-07-28 09:24:37,894 [myid:] - INFO [QuorumPeer[myid=2](plain=/127.0.0.1:11225)(secure=disabled):QuorumPeer@1033] - LOOKING
2016-07-28 09:24:37,895 [myid:] - INFO [QuorumPeer[myid=2](plain=/127.0.0.1:11225)(secure=disabled):FastLeaderElection@894] - New election. My id = 2, proposed zxid=0x0
2016-07-28 09:24:37,895 [myid:] - INFO [QuorumPeer[myid=1](plain=/127.0.0.1:11222)(secure=disabled):FastLeaderElection@894] - New election. My id = 1, proposed zxid=0x0
2016-07-28 09:24:37,899 [myid:] - INFO [/127.0.0.1:11227:QuorumCnxManager$Listener@638] - Received connection request /127.0.0.1:50183
2016-07-28 09:24:37,899 [myid:] - INFO [WorkerSender[myid=1]:QuorumCnxManager@276] - Have smaller server identifier, so dropping the connection: (2, 1)
2016-07-28 09:24:37,901 [myid:] - INFO [/127.0.0.1:11224:QuorumCnxManager$Listener@638] - Received connection request /127.0.0.1:50185
2016-07-28 09:24:37,901 [myid:] - WARN [WorkerSender[myid=1]:QuorumCnxManager@455] - Cannot open channel to 3 at election address /127.0.0.1:11230
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:441)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:482)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:419)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:486)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:465)
at java.lang.Thread.run(Thread.java:745)
2016-07-28 09:24:37,905 [myid:] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@688] - Notification: 2 (message format version), 1 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version)
2016-07-28 09:24:37,906 [myid:] - WARN [WorkerSender[myid=2]:QuorumCnxManager@455] - Cannot open channel to 3 at election address /127.0.0.1:11230
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:441)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:482)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:419)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:486)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:465)
at java.lang.Thread.run(Thread.java:745)
2016-07-28 09:24:37,906 [myid:] - WARN [WorkerSender[myid=2]:QuorumCnxManager@455] - Cannot open channel to 4 at election address /127.0.0.1:11233
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:441)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:482)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:419)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:486)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:465)
at java.lang.Thread.run(Thread.java:745)
2016-07-28 09:24:37,905 [myid:] - WARN [WorkerSender[myid=1]:QuorumCnxManager@455] - Cannot open channel to 4 at election address /127.0.0.1:11233
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:441)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:482)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:419)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:486)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:465)
at java.lang.Thread.run(Thread.java:745)
2016-07-28 09:24:37,907 [myid:] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@688] - Notification: 2 (message format version), 1 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version)
2016-07-28 09:24:37,908 [myid:] - WARN [WorkerSender[myid=1]:QuorumCnxManager@455] - Cannot open channel to 5 at election address /127.0.0.1:11236
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:441)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:482)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:419)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:486)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:465)
at java.lang.Thread.run(Thread.java:745)
2016-07-28 09:24:37,920 [myid:] - INFO [main:QuorumPeer@791] - acceptedEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
2016-07-28 09:24:37,961 [myid:] - WARN [WorkerSender[myid=2]:QuorumCnxManager@455] - Cannot open channel to 5 at election address /127.0.0.1:11236
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:441)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:482)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:419)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:486)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:465)
at java.lang.Thread.run(Thread.java:745)
2016-07-28 09:24:37,908 [myid:] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@688] - Notification: 2 (message format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version)
2016-07-28 09:24:37,962 [myid:] - WARN [WorkerSender[myid=1]:QuorumCnxManager@455] - Cannot open channel to 6 at election address /127.0.0.1:11239
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:441)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:482)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:419)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:486)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:465)
at java.lang.Thread.run(Thread.java:745)
2016-07-28 09:24:37,962 [myid:] - WARN [WorkerSender[myid=2]:QuorumCnxManager@455] - Cannot open channel to 6 at election address /127.0.0.1:11239
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:441)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:482)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:419)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:486)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:465)
at java.lang.Thread.run(Thread.java:745)
2016-07-28 09:24:37,963 [myid:] - WARN [WorkerSender[myid=1]:QuorumCnxManager@455] - Cannot open channel to 7 at election address /127.0.0.1:11242
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:441)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:482)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:419)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:486)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:465)
at java.lang.Thread.run(Thread.java:745)
2016-07-28 09:24:37,963 [myid:] - WARN [WorkerSender[myid=2]:QuorumCnxManager@455] - Cannot open channel to 7 at election address /127.0.0.1:11242
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:441)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:482)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:419)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:486)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:465)
at java.lang.Thread.run(Thread.java:745)
2016-07-28 09:24:37,965 [myid:] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@688] - Notification: 2 (message format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version)
2016-07-28 09:24:37,967 [myid:] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@688] - Notification: 2 (message format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version)
2016-07-28 09:24:37,968 [myid:] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@688] - Notification: 2 (message format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version)
2016-07-28 09:24:37,968 [myid:] - WARN [WorkerSender[myid=1]:QuorumCnxManager@455] - Cannot open channel to 3 at election address /127.0.0.1:11230
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:441)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:482)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:419)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:486)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:465)
at java.lang.Thread.run(Thread.java:745)
2016-07-28 09:24:37,969 [myid:] - WARN [WorkerSender[myid=1]:QuorumCnxManager@455] - Cannot open channel to 4 at election address /127.0.0.1:11233
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:441)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:482)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:419)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:486)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:465)
at java.lang.Thread.run(Thread.java:745)
2016-07-28 09:24:37,969 [myid:] - WARN [WorkerSender[myid=1]:QuorumCnxManager@455] - Cannot open channel to 5 at election address /127.0.0.1:11236
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:441)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:482)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:419)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:486)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:465)
at java.lang.Thread.run(Thread.java:745)
2016-07-28 09:24:37,981 [myid:] - WARN [WorkerSender[myid=1]:QuorumCnxManager@455] - Cannot open channel to 6 at election address /127.0.0.1:11239
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:441)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:482)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:419)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:486)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:465)
at java.lang.Thread.run(Thread.java:745)
2016-07-28 09:24:37,982 [myid:] - WARN [WorkerSender[myid=1]:QuorumCnxManager@455] - Cannot open channel to 7 at election address /127.0.0.1:11242
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:441)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:482)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:419)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:486)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:465)
at java.lang.Thread.run(Thread.java:745)
2016-07-28 09:24:38,015 [myid:] - INFO [main:QuorumUtil@146] - Started QuorumPeer 3
2016-07-28 09:24:38,016 [myid:] - INFO [main:QuorumUtil@203] - Creating QuorumPeer 4; public port 11231
2016-07-28 09:24:38,016 [myid:] - INFO [main:NIOServerCnxnFactory@673] - Configuring NIO connection handler with 10s sessionless connection timeout, 2 selector thread(s), 16 worker threads, and 64 kB direct buffers.
2016-07-28 09:24:38,017 [myid:] - INFO [main:NIOServerCnxnFactory@686] - binding to port /127.0.0.1:11231
2016-07-28 09:24:38,017 [myid:] - INFO [QuorumPeer[myid=3](plain=/127.0.0.1:11228)(secure=disabled):QuorumPeer@1033] - LOOKING
2016-07-28 09:24:38,018 [myid:] - INFO [QuorumPeer[myid=3](plain=/127.0.0.1:11228)(secure=disabled):FastLeaderElection@894] - New election. My id = 3, proposed zxid=0x0
2016-07-28 09:24:38,018 [myid:] - INFO [main:QuorumPeer@776] - currentEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
2016-07-28 09:24:38,020 [myid:] - INFO [QuorumPeerListener:QuorumCnxManager$Listener@632] - My election bind port: /127.0.0.1:11230
2016-07-28 09:24:38,025 [myid:] - INFO [/127.0.0.1:11224:QuorumCnxManager$Listener@638] - Received connection request /127.0.0.1:50200
2016-07-28 09:24:38,026 [myid:] - INFO [/127.0.0.1:11227:QuorumCnxManager$Listener@638] - Received connection request /127.0.0.1:50201
2016-07-28 09:24:38,027 [myid:] - WARN [WorkerSender[myid=3]:QuorumCnxManager@455] - Cannot open channel to 4 at election address /127.0.0.1:11233
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:441)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:482)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:419)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:486)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:465)
at java.lang.Thread.run(Thread.java:745)
2016-07-28 09:24:38,027 [myid:] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@688] - Notification: 2 (message format version), 3 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 3 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version)
2016-07-28 09:24:38,028 [myid:] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@688] - Notification: 2 (message format version), 3 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 3 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version)
2016-07-28 09:24:38,029 [myid:] - INFO [WorkerReceiver[myid=3]:FastLeaderElection@688] - Notification: 2 (message format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version)
2016-07-28 09:24:38,029 [myid:] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@688] - Notification: 2 (message format version), 3 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version)
2016-07-28 09:24:38,031 [myid:] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@688] - Notification: 2 (message format version), 3 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version)
2016-07-28 09:24:38,031 [myid:] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@688] - Notification: 2 (message format version), 3 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version)
2016-07-28 09:24:38,031 [myid:] - INFO [WorkerReceiver[myid=3]:FastLeaderElection@688] - Notification: 2 (message format version), 3 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 3 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version)
2016-07-28 09:24:38,032 [myid:] - WARN [WorkerSender[myid=1]:QuorumCnxManager@455] - Cannot open channel to 4 at election address /127.0.0.1:11233
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:441)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:482)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:419)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:486)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:465)
at java.lang.Thread.run(Thread.java:745)
2016-07-28 09:24:38,032 [myid:] - WARN [WorkerSender[myid=1]:QuorumCnxManager@455] - Cannot open channel to 5 at election address /127.0.0.1:11236
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:441)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:482)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:419)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:486)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:465)
at java.lang.Thread.run(Thread.java:745)
2016-07-28 09:24:38,033 [myid:] - WARN [WorkerSender[myid=1]:QuorumCnxManager@455] - Cannot open channel to 6 at election address /127.0.0.1:11239
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:441)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:482)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:419)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:486)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:465)
at java.lang.Thread.run(Thread.java:745)
2016-07-28 09:24:38,033 [myid:] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@688] - Notification: 2 (message format version), 3 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version)
2016-07-28 09:24:38,032 [myid:] - INFO [WorkerReceiver[myid=3]:FastLeaderElection@688] - Notification: 2 (message format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version)
2016-07-28 09:24:38,033 [myid:] - WARN [WorkerSender[myid=2]:QuorumCnxManager@455] - Cannot open channel to 4 at election address /127.0.0.1:11233
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:441)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:482)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:419)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:486)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:465)
at java.lang.Thread.run(Thread.java:745)
2016-07-28 09:24:38,034 [myid:] - WARN [WorkerSender[myid=1]:QuorumCnxManager@455] - Cannot open channel to 7 at election address /127.0.0.1:11242
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:441)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:482)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:419)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:486)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:465)
at java.lang.Thread.run(Thread.java:745)
2016-07-28 09:24:38,035 [myid:] - INFO [WorkerReceiver[myid=3]:FastLeaderElection@688] - Notification: 2 (message format version), 3 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version)
2016-07-28 09:24:38,036 [myid:] - WARN [WorkerSender[myid=2]:QuorumCnxManager@455] - Cannot open channel to 5 at election address /127.0.0.1:11236
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:441)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:482)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:419)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:486)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:465)
at java.lang.Thread.run(Thread.java:745)
2016-07-28 09:24:38,037 [myid:] - INFO [WorkerReceiver[myid=3]:FastLeaderElection@688] - Notification: 2 (message format version), 3 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version)
2016-07-28 09:24:38,037 [myid:] - WARN [WorkerSender[myid=2]:QuorumCnxManager@455] - Cannot open channel to 6 at election address /127.0.0.1:11239
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:441)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:482)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:419)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:486)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:465)
at java.lang.Thread.run(Thread.java:745)
2016-07-28 09:24:38,038 [myid:] - WARN [WorkerSender[myid=2]:QuorumCnxManager@455] - Cannot open channel to 7 at election address /127.0.0.1:11242
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:441)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:482)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:419)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:486)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:465)
at java.lang.Thread.run(Thread.java:745)
2016-07-28 09:24:38,044 [myid:] - WARN [WorkerSender[myid=3]:QuorumCnxManager@455] - Cannot open channel to 5 at election address /127.0.0.1:11236
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:441)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:482)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:419)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:486)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:465)
at java.lang.Thread.run(Thread.java:745)
2016-07-28 09:24:38,045 [myid:] - WARN [WorkerSender[myid=3]:QuorumCnxManager@455] - Cannot open channel to 6 at election address /127.0.0.1:11239
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:441)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:482)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:419)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:486)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:465)
at java.lang.Thread.run(Thread.java:745)
2016-07-28 09:24:38,045 [myid:] - WARN [WorkerSender[myid=3]:QuorumCnxManager@455] - Cannot open channel to 7 at election address /127.0.0.1:11242
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)

...[truncated 3451538 chars]...
mPeer[myid=1](plain=/127.0.0.1:11371)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id1]
2016-07-28 09:30:21,332 [myid:] - INFO [QuorumPeer[myid=1](plain=/127.0.0.1:11371)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id1,name1=replica.1]
2016-07-28 09:30:21,332 [myid:] - INFO [QuorumPeer[myid=1](plain=/127.0.0.1:11371)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id1,name1=replica.2]
2016-07-28 09:30:21,332 [myid:] - INFO [QuorumPeer[myid=1](plain=/127.0.0.1:11371)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id1,name1=replica.3]
2016-07-28 09:30:21,332 [myid:] - INFO [QuorumPeer[myid=1](plain=/127.0.0.1:11371)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id1,name1=replica.5]
2016-07-28 09:30:21,332 [myid:] - INFO [main:QuorumUtil@250] - Shutting down quorum peer QuorumPeer[myid=2](plain=/127.0.0.1:11374)(secure=disabled)
2016-07-28 09:30:21,332 [myid:] - INFO [main:Follower@198] - shutdown called
java.lang.Exception: shutdown Follower
at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:198)
at org.apache.zookeeper.server.quorum.QuorumPeer.shutdown(QuorumPeer.java:1184)
at org.apache.zookeeper.test.QuorumUtil.shutdown(QuorumUtil.java:251)
at org.apache.zookeeper.test.QuorumUtil.shutdownAll(QuorumUtil.java:238)
at org.apache.zookeeper.test.QuorumUtil.tearDown(QuorumUtil.java:306)
at org.apache.zookeeper.test.ReconfigTest.tearDown(ReconfigTest.java:64)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33)
at org.junit.rules.TestWatchman$1.evaluate(TestWatchman.java:53)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:38)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:518)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1052)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:906)
2016-07-28 09:30:21,333 [myid:] - INFO [main:LearnerZooKeeperServer@165] - Shutting down
2016-07-28 09:30:21,333 [myid:] - INFO [main:ZooKeeperServer@498] - shutting down
2016-07-28 09:30:21,332 [myid:] - INFO [QuorumPeer[myid=2](plain=/127.0.0.1:11374)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id2,name1=replica.2,name2=Follower]
2016-07-28 09:30:21,333 [myid:] - INFO [main:FollowerRequestProcessor@138] - Shutting down
2016-07-28 09:30:21,333 [myid:] - INFO [QuorumPeer[myid=2](plain=/127.0.0.1:11374)(secure=disabled):Follower@198] - shutdown called
java.lang.Exception: shutdown Follower
at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:198)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1115)
2016-07-28 09:30:21,333 [myid:] - INFO [FollowerRequestProcessor:2:FollowerRequestProcessor@109] - FollowerRequestProcessor exited loop!
2016-07-28 09:30:21,333 [myid:] - INFO [main:CommitProcessor@414] - Shutting down
2016-07-28 09:30:21,334 [myid:] - INFO [CommitProcessor:2:CommitProcessor@299] - CommitProcessor exited loop!
2016-07-28 09:30:21,334 [myid:] - INFO [main:FinalRequestProcessor@479] - shutdown of request processor complete
2016-07-28 09:30:21,334 [myid:] - INFO [main:MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id2,name1=replica.2,name2=Follower,name3=InMemoryDataTree]
2016-07-28 09:30:21,334 [myid:] - INFO [main:SyncRequestProcessor@191] - Shutting down
2016-07-28 09:30:21,334 [myid:] - INFO [SyncThread:2:SyncRequestProcessor@169] - SyncRequestProcessor exited!
2016-07-28 09:30:21,334 [myid:] - WARN [QuorumPeer[myid=2](plain=/127.0.0.1:11374)(secure=disabled):QuorumPeer@1158] - PeerState set to LOOKING
2016-07-28 09:30:21,335 [myid:] - WARN [QuorumPeer[myid=2](plain=/127.0.0.1:11374)(secure=disabled):QuorumPeer@1140] - QuorumPeer main thread exited
2016-07-28 09:30:21,335 [myid:] - INFO [QuorumPeer[myid=2](plain=/127.0.0.1:11374)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id2]
2016-07-28 09:30:21,335 [myid:] - INFO [ConnnectionExpirer:NIOServerCnxnFactory$ConnectionExpirerThread@583] - ConnnectionExpirerThread interrupted
2016-07-28 09:30:21,335 [myid:] - INFO [QuorumPeer[myid=2](plain=/127.0.0.1:11374)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id2,name1=replica.2]
2016-07-28 09:30:21,336 [myid:] - INFO [NIOServerCxnFactory.AcceptThread:/127.0.0.1:11374:NIOServerCnxnFactory$AcceptThread@219] - accept thread exitted run method
2016-07-28 09:30:21,337 [myid:] - INFO [QuorumPeer[myid=2](plain=/127.0.0.1:11374)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id2,name1=replica.1]
2016-07-28 09:30:21,337 [myid:] - INFO [QuorumPeer[myid=2](plain=/127.0.0.1:11374)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id2,name1=replica.3]
2016-07-28 09:30:21,337 [myid:] - INFO [NIOServerCxnFactory.SelectorThread-0:NIOServerCnxnFactory$SelectorThread@420] - selector thread exitted run method
2016-07-28 09:30:21,337 [myid:] - INFO [QuorumPeer[myid=2](plain=/127.0.0.1:11374)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id2,name1=replica.5]
2016-07-28 09:30:21,337 [myid:] - INFO [NIOServerCxnFactory.SelectorThread-1:NIOServerCnxnFactory$SelectorThread@420] - selector thread exitted run method
2016-07-28 09:30:21,337 [myid:] - INFO [/127.0.0.1:11376:QuorumCnxManager$Listener@661] - Leaving listener
2016-07-28 09:30:21,338 [myid:] - INFO [main:QuorumUtil@254] - Shutting down leader election QuorumPeer[myid=2](plain=/127.0.0.1:11374)(secure=disabled)
2016-07-28 09:30:21,339 [myid:] - INFO [main:QuorumUtil@259] - Waiting for QuorumPeer[myid=2](plain=/127.0.0.1:11374)(secure=disabled) to exit thread
2016-07-28 09:30:21,339 [myid:] - INFO [main:QuorumUtil@250] - Shutting down quorum peer QuorumPeer[myid=3](plain=/127.0.0.1:11377)(secure=disabled)
2016-07-28 09:30:21,339 [myid:] - INFO [main:Follower@198] - shutdown called
java.lang.Exception: shutdown Follower
at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:198)
at org.apache.zookeeper.server.quorum.QuorumPeer.shutdown(QuorumPeer.java:1184)
at org.apache.zookeeper.test.QuorumUtil.shutdown(QuorumUtil.java:251)
at org.apache.zookeeper.test.QuorumUtil.shutdownAll(QuorumUtil.java:238)
at org.apache.zookeeper.test.QuorumUtil.tearDown(QuorumUtil.java:306)
at org.apache.zookeeper.test.ReconfigTest.tearDown(ReconfigTest.java:64)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33)
at org.junit.rules.TestWatchman$1.evaluate(TestWatchman.java:53)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:38)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:518)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1052)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:906)
2016-07-28 09:30:21,339 [myid:] - INFO [main:LearnerZooKeeperServer@165] - Shutting down
2016-07-28 09:30:21,339 [myid:] - INFO [main:ZooKeeperServer@498] - shutting down
2016-07-28 09:30:21,339 [myid:] - INFO [main:FollowerRequestProcessor@138] - Shutting down
2016-07-28 09:30:21,339 [myid:] - INFO [main:CommitProcessor@414] - Shutting down
2016-07-28 09:30:21,339 [myid:] - INFO [FollowerRequestProcessor:3:FollowerRequestProcessor@109] - FollowerRequestProcessor exited loop!
2016-07-28 09:30:21,339 [myid:] - INFO [CommitProcessor:3:CommitProcessor@299] - CommitProcessor exited loop!
2016-07-28 09:30:21,339 [myid:] - INFO [main:FinalRequestProcessor@479] - shutdown of request processor complete
2016-07-28 09:30:21,340 [myid:] - INFO [main:MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id3,name1=replica.3,name2=Follower,name3=InMemoryDataTree]
2016-07-28 09:30:21,340 [myid:] - INFO [main:SyncRequestProcessor@191] - Shutting down
2016-07-28 09:30:21,340 [myid:] - INFO [SyncThread:3:SyncRequestProcessor@169] - SyncRequestProcessor exited!
2016-07-28 09:30:21,340 [myid:] - INFO [ConnnectionExpirer:NIOServerCnxnFactory$ConnectionExpirerThread@583] - ConnnectionExpirerThread interrupted
2016-07-28 09:30:21,341 [myid:] - INFO [NIOServerCxnFactory.SelectorThread-0:NIOServerCnxnFactory$SelectorThread@420] - selector thread exitted run method
2016-07-28 09:30:21,341 [myid:] - INFO [NIOServerCxnFactory.AcceptThread:/127.0.0.1:11377:NIOServerCnxnFactory$AcceptThread@219] - accept thread exitted run method
2016-07-28 09:30:21,342 [myid:] - INFO [NIOServerCxnFactory.SelectorThread-1:NIOServerCnxnFactory$SelectorThread@420] - selector thread exitted run method
2016-07-28 09:30:21,342 [myid:] - INFO [/127.0.0.1:11379:QuorumCnxManager$Listener@661] - Leaving listener
2016-07-28 09:30:21,343 [myid:] - INFO [main:QuorumUtil@254] - Shutting down leader election QuorumPeer[myid=3](plain=/127.0.0.1:11377)(secure=disabled)
2016-07-28 09:30:21,343 [myid:] - INFO [main:QuorumUtil@259] - Waiting for QuorumPeer[myid=3](plain=/127.0.0.1:11377)(secure=disabled) to exit thread
2016-07-28 09:30:21,421 [myid:127.0.0.1:11273] - INFO [main-SendThread(127.0.0.1:11273):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:11273. Will not attempt to authenticate using SASL (unknown error)
2016-07-28 09:30:21,422 [myid:127.0.0.1:11273] - ERROR [main-SendThread(127.0.0.1:11273):ClientCnxnSocketNIO@287] - Unable to open socket to 127.0.0.1/127.0.0.1:11273
2016-07-28 09:30:21,422 [myid:127.0.0.1:11273] - WARN [main-SendThread(127.0.0.1:11273):ClientCnxn$SendThread@1235] - Session 0x222add090c60000 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.Net.connect0(Native Method)
at sun.nio.ch.Net.connect(Net.java:465)
at sun.nio.ch.Net.connect(Net.java:457)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670)
at org.apache.zookeeper.ClientCnxnSocketNIO.registerAndConnect(ClientCnxnSocketNIO.java:275)
at org.apache.zookeeper.ClientCnxnSocketNIO.connect(ClientCnxnSocketNIO.java:285)
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1098)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1133)
2016-07-28 09:30:21,451 [myid:127.0.0.1:11257] - INFO [main-SendThread(127.0.0.1:11257):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:11257. Will not attempt to authenticate using SASL (unknown error)
2016-07-28 09:30:21,451 [myid:127.0.0.1:11257] - ERROR [main-SendThread(127.0.0.1:11257):ClientCnxnSocketNIO@287] - Unable to open socket to 127.0.0.1/127.0.0.1:11257
2016-07-28 09:30:21,451 [myid:127.0.0.1:11257] - WARN [main-SendThread(127.0.0.1:11257):ClientCnxn$SendThread@1235] - Session 0x122adcf0ae70000 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.Net.connect0(Native Method)
at sun.nio.ch.Net.connect(Net.java:465)
at sun.nio.ch.Net.connect(Net.java:457)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670)
at org.apache.zookeeper.ClientCnxnSocketNIO.registerAndConnect(ClientCnxnSocketNIO.java:275)
at org.apache.zookeeper.ClientCnxnSocketNIO.connect(ClientCnxnSocketNIO.java:285)
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1098)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1133)
2016-07-28 09:30:21,671 [myid:127.0.0.1:11276] - INFO [main-SendThread(127.0.0.1:11276):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:11276. Will not attempt to authenticate using SASL (unknown error)
2016-07-28 09:30:21,671 [myid:127.0.0.1:11276] - ERROR [main-SendThread(127.0.0.1:11276):ClientCnxnSocketNIO@287] - Unable to open socket to 127.0.0.1/127.0.0.1:11276
2016-07-28 09:30:21,671 [myid:127.0.0.1:11276] - WARN [main-SendThread(127.0.0.1:11276):ClientCnxn$SendThread@1235] - Session 0x322add08d0d0000 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.Net.connect0(Native Method)
at sun.nio.ch.Net.connect(Net.java:465)
at sun.nio.ch.Net.connect(Net.java:457)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670)
at org.apache.zookeeper.ClientCnxnSocketNIO.registerAndConnect(ClientCnxnSocketNIO.java:275)
at org.apache.zookeeper.ClientCnxnSocketNIO.connect(ClientCnxnSocketNIO.java:285)
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1098)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1133)
2016-07-28 09:30:21,991 [myid:127.0.0.1:11306] - INFO [main-SendThread(127.0.0.1:11306):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:11306. Will not attempt to authenticate using SASL (unknown error)
2016-07-28 09:30:21,991 [myid:127.0.0.1:11306] - ERROR [main-SendThread(127.0.0.1:11306):ClientCnxnSocketNIO@287] - Unable to open socket to 127.0.0.1/127.0.0.1:11306
2016-07-28 09:30:21,991 [myid:127.0.0.1:11306] - WARN [main-SendThread(127.0.0.1:11306):ClientCnxn$SendThread@1235] - Session 0x222add109d00000 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.Net.connect0(Native Method)
at sun.nio.ch.Net.connect(Net.java:465)
at sun.nio.ch.Net.connect(Net.java:457)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670)
at org.apache.zookeeper.ClientCnxnSocketNIO.registerAndConnect(ClientCnxnSocketNIO.java:275)
at org.apache.zookeeper.ClientCnxnSocketNIO.connect(ClientCnxnSocketNIO.java:285)
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1098)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1133)
2016-07-28 09:30:22,091 [myid:127.0.0.1:11263] - INFO [main-SendThread(127.0.0.1:11263):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:11263. Will not attempt to authenticate using SASL (unknown error)
2016-07-28 09:30:22,091 [myid:127.0.0.1:11263] - ERROR [main-SendThread(127.0.0.1:11263):ClientCnxnSocketNIO@287] - Unable to open socket to 127.0.0.1/127.0.0.1:11263
2016-07-28 09:30:22,091 [myid:127.0.0.1:11263] - WARN [main-SendThread(127.0.0.1:11263):ClientCnxn$SendThread@1235] - Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.Net.connect0(Native Method)
at sun.nio.ch.Net.connect(Net.java:465)
at sun.nio.ch.Net.connect(Net.java:457)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670)
at org.apache.zookeeper.ClientCnxnSocketNIO.registerAndConnect(ClientCnxnSocketNIO.java:275)
at org.apache.zookeeper.ClientCnxnSocketNIO.connect(ClientCnxnSocketNIO.java:285)
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1098)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1133)
2016-07-28 09:30:22,111 [myid:127.0.0.1:11303] - INFO [main-SendThread(127.0.0.1:11303):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:11303. Will not attempt to authenticate using SASL (unknown error)
2016-07-28 09:30:22,111 [myid:127.0.0.1:11303] - ERROR [main-SendThread(127.0.0.1:11303):ClientCnxnSocketNIO@287] - Unable to open socket to 127.0.0.1/127.0.0.1:11303
2016-07-28 09:30:22,111 [myid:127.0.0.1:11303] - WARN [main-SendThread(127.0.0.1:11303):ClientCnxn$SendThread@1235] - Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.Net.connect0(Native Method)
at sun.nio.ch.Net.connect(Net.java:465)
at sun.nio.ch.Net.connect(Net.java:457)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670)
at org.apache.zookeeper.ClientCnxnSocketNIO.registerAndConnect(ClientCnxnSocketNIO.java:275)
at org.apache.zookeeper.ClientCnxnSocketNIO.connect(ClientCnxnSocketNIO.java:285)
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1098)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1133)
2016-07-28 09:30:22,241 [myid:127.0.0.1:11270] - INFO [main-SendThread(127.0.0.1:11270):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:11270. Will not attempt to authenticate using SASL (unknown error)
2016-07-28 09:30:22,241 [myid:127.0.0.1:11270] - ERROR [main-SendThread(127.0.0.1:11270):ClientCnxnSocketNIO@287] - Unable to open socket to 127.0.0.1/127.0.0.1:11270
2016-07-28 09:30:22,241 [myid:127.0.0.1:11270] - WARN [main-SendThread(127.0.0.1:11270):ClientCnxn$SendThread@1235] - Session 0x122add08ccf0000 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.Net.connect0(Native Method)
at sun.nio.ch.Net.connect(Net.java:465)
at sun.nio.ch.Net.connect(Net.java:457)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670)
at org.apache.zookeeper.ClientCnxnSocketNIO.registerAndConnect(ClientCnxnSocketNIO.java:275)
at org.apache.zookeeper.ClientCnxnSocketNIO.connect(ClientCnxnSocketNIO.java:285)
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1098)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1133)
2016-07-28 09:30:22,341 [myid:] - INFO [QuorumPeer[myid=3](plain=/127.0.0.1:11377)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id3,name1=replica.3,name2=Follower]
2016-07-28 09:30:22,341 [myid:] - INFO [QuorumPeer[myid=3](plain=/127.0.0.1:11377)(secure=disabled):Follower@198] - shutdown called
java.lang.Exception: shutdown Follower
at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:198)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1115)
2016-07-28 09:30:22,341 [myid:] - WARN [QuorumPeer[myid=3](plain=/127.0.0.1:11377)(secure=disabled):QuorumPeer@1158] - PeerState set to LOOKING
2016-07-28 09:30:22,341 [myid:] - WARN [QuorumPeer[myid=3](plain=/127.0.0.1:11377)(secure=disabled):QuorumPeer@1140] - QuorumPeer main thread exited
2016-07-28 09:30:22,341 [myid:] - INFO [QuorumPeer[myid=3](plain=/127.0.0.1:11377)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id3]
2016-07-28 09:30:22,342 [myid:] - INFO [QuorumPeer[myid=3](plain=/127.0.0.1:11377)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id3,name1=replica.3]
2016-07-28 09:30:22,342 [myid:] - INFO [QuorumPeer[myid=3](plain=/127.0.0.1:11377)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id3,name1=replica.1]
2016-07-28 09:30:22,342 [myid:] - INFO [QuorumPeer[myid=3](plain=/127.0.0.1:11377)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id3,name1=replica.2]
2016-07-28 09:30:22,342 [myid:] - INFO [QuorumPeer[myid=3](plain=/127.0.0.1:11377)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id3,name1=replica.5]
2016-07-28 09:30:22,342 [myid:] - INFO [main:QuorumUtil@250] - Shutting down quorum peer QuorumPeer[myid=4](plain=/127.0.0.1:11380)(secure=disabled)
2016-07-28 09:30:22,342 [myid:] - INFO [main:Follower@198] - shutdown called
java.lang.Exception: shutdown Follower
at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:198)
at org.apache.zookeeper.server.quorum.QuorumPeer.shutdown(QuorumPeer.java:1184)
at org.apache.zookeeper.test.QuorumUtil.shutdown(QuorumUtil.java:251)
at org.apache.zookeeper.test.QuorumUtil.shutdownAll(QuorumUtil.java:238)
at org.apache.zookeeper.test.QuorumUtil.tearDown(QuorumUtil.java:306)
at org.apache.zookeeper.test.ReconfigTest.tearDown(ReconfigTest.java:64)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33)
at org.junit.rules.TestWatchman$1.evaluate(TestWatchman.java:53)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:38)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:518)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1052)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:906)
2016-07-28 09:30:22,342 [myid:] - INFO [main:LearnerZooKeeperServer@165] - Shutting down
2016-07-28 09:30:22,342 [myid:] - INFO [main:ZooKeeperServer@498] - shutting down
2016-07-28 09:30:22,343 [myid:] - INFO [main:FollowerRequestProcessor@138] - Shutting down
2016-07-28 09:30:22,343 [myid:] - INFO [main:CommitProcessor@414] - Shutting down
2016-07-28 09:30:22,343 [myid:] - INFO [FollowerRequestProcessor:4:FollowerRequestProcessor@109] - FollowerRequestProcessor exited loop!
2016-07-28 09:30:22,343 [myid:] - INFO [CommitProcessor:4:CommitProcessor@299] - CommitProcessor exited loop!
2016-07-28 09:30:22,343 [myid:] - INFO [main:FinalRequestProcessor@479] - shutdown of request processor complete
2016-07-28 09:30:22,344 [myid:] - INFO [main:MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id4,name1=replica.4,name2=Follower,name3=InMemoryDataTree]
2016-07-28 09:30:22,344 [myid:] - INFO [main:SyncRequestProcessor@191] - Shutting down
2016-07-28 09:30:22,344 [myid:] - INFO [SyncThread:4:SyncRequestProcessor@169] - SyncRequestProcessor exited!
2016-07-28 09:30:22,344 [myid:] - INFO [ConnnectionExpirer:NIOServerCnxnFactory$ConnectionExpirerThread@583] - ConnnectionExpirerThread interrupted
2016-07-28 09:30:22,345 [myid:] - INFO [NIOServerCxnFactory.AcceptThread:/127.0.0.1:11380:NIOServerCnxnFactory$AcceptThread@219] - accept thread exitted run method
2016-07-28 09:30:22,345 [myid:] - INFO [NIOServerCxnFactory.SelectorThread-0:NIOServerCnxnFactory$SelectorThread@420] - selector thread exitted run method
2016-07-28 09:30:22,345 [myid:] - INFO [NIOServerCxnFactory.SelectorThread-1:NIOServerCnxnFactory$SelectorThread@420] - selector thread exitted run method
2016-07-28 09:30:22,346 [myid:] - INFO [/127.0.0.1:11382:QuorumCnxManager$Listener@661] - Leaving listener
2016-07-28 09:30:22,346 [myid:] - INFO [main:QuorumUtil@254] - Shutting down leader election QuorumPeer[myid=4](plain=/127.0.0.1:11380)(secure=disabled)
2016-07-28 09:30:22,346 [myid:] - INFO [main:QuorumUtil@259] - Waiting for QuorumPeer[myid=4](plain=/127.0.0.1:11380)(secure=disabled) to exit thread
2016-07-28 09:30:22,521 [myid:127.0.0.1:11309] - INFO [main-SendThread(127.0.0.1:11309):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:11309. Will not attempt to authenticate using SASL (unknown error)
2016-07-28 09:30:22,521 [myid:127.0.0.1:11309] - ERROR [main-SendThread(127.0.0.1:11309):ClientCnxnSocketNIO@287] - Unable to open socket to 127.0.0.1/127.0.0.1:11309
2016-07-28 09:30:22,521 [myid:127.0.0.1:11309] - WARN [main-SendThread(127.0.0.1:11309):ClientCnxn$SendThread@1235] - Session 0x322add109d50000 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.Net.connect0(Native Method)
at sun.nio.ch.Net.connect(Net.java:465)
at sun.nio.ch.Net.connect(Net.java:457)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670)
at org.apache.zookeeper.ClientCnxnSocketNIO.registerAndConnect(ClientCnxnSocketNIO.java:275)
at org.apache.zookeeper.ClientCnxnSocketNIO.connect(ClientCnxnSocketNIO.java:285)
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1098)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1133)
2016-07-28 09:30:22,631 [myid:127.0.0.1:11273] - INFO [main-SendThread(127.0.0.1:11273):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:11273. Will not attempt to authenticate using SASL (unknown error)
2016-07-28 09:30:22,631 [myid:127.0.0.1:11273] - ERROR [main-SendThread(127.0.0.1:11273):ClientCnxnSocketNIO@287] - Unable to open socket to 127.0.0.1/127.0.0.1:11273
2016-07-28 09:30:22,631 [myid:127.0.0.1:11273] - WARN [main-SendThread(127.0.0.1:11273):ClientCnxn$SendThread@1235] - Session 0x222add090c60000 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.Net.connect0(Native Method)
at sun.nio.ch.Net.connect(Net.java:465)
at sun.nio.ch.Net.connect(Net.java:457)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670)
at org.apache.zookeeper.ClientCnxnSocketNIO.registerAndConnect(ClientCnxnSocketNIO.java:275)
at org.apache.zookeeper.ClientCnxnSocketNIO.connect(ClientCnxnSocketNIO.java:285)
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1098)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1133)
2016-07-28 09:30:22,761 [myid:] - INFO [WorkerReceiver[myid=4]:FastLeaderElection$Messenger$WorkerReceiver@440] - WorkerReceiver is down
2016-07-28 09:30:22,761 [myid:] - INFO [WorkerSender[myid=4]:FastLeaderElection$Messenger$WorkerSender@470] - WorkerSender is down
2016-07-28 09:30:22,771 [myid:] - INFO [WorkerReceiver[myid=1]:FastLeaderElection$Messenger$WorkerReceiver@440] - WorkerReceiver is down
2016-07-28 09:30:22,771 [myid:] - INFO [WorkerSender[myid=1]:FastLeaderElection$Messenger$WorkerSender@470] - WorkerSender is down
2016-07-28 09:30:22,771 [myid:] - INFO [WorkerSender[myid=2]:FastLeaderElection$Messenger$WorkerSender@470] - WorkerSender is down
2016-07-28 09:30:22,771 [myid:] - INFO [WorkerReceiver[myid=3]:FastLeaderElection$Messenger$WorkerReceiver@440] - WorkerReceiver is down
2016-07-28 09:30:22,771 [myid:] - INFO [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@440] - WorkerReceiver is down
2016-07-28 09:30:22,771 [myid:] - INFO [WorkerSender[myid=3]:FastLeaderElection$Messenger$WorkerSender@470] - WorkerSender is down
2016-07-28 09:30:23,061 [myid:127.0.0.1:11260] - INFO [main-SendThread(127.0.0.1:11260):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:11260. Will not attempt to authenticate using SASL (unknown error)
2016-07-28 09:30:23,061 [myid:127.0.0.1:11260] - ERROR [main-SendThread(127.0.0.1:11260):ClientCnxnSocketNIO@287] - Unable to open socket to 127.0.0.1/127.0.0.1:11260
2016-07-28 09:30:23,061 [myid:127.0.0.1:11260] - WARN [main-SendThread(127.0.0.1:11260):ClientCnxn$SendThread@1235] - Session 0x222adcf0da20000 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.Net.connect0(Native Method)
at sun.nio.ch.Net.connect(Net.java:465)
at sun.nio.ch.Net.connect(Net.java:457)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670)
at org.apache.zookeeper.ClientCnxnSocketNIO.registerAndConnect(ClientCnxnSocketNIO.java:275)
at org.apache.zookeeper.ClientCnxnSocketNIO.connect(ClientCnxnSocketNIO.java:285)
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1098)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1133)
2016-07-28 09:30:23,211 [myid:127.0.0.1:11263] - INFO [main-SendThread(127.0.0.1:11263):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:11263. Will not attempt to authenticate using SASL (unknown error)
2016-07-28 09:30:23,211 [myid:127.0.0.1:11263] - ERROR [main-SendThread(127.0.0.1:11263):ClientCnxnSocketNIO@287] - Unable to open socket to 127.0.0.1/127.0.0.1:11263
2016-07-28 09:30:23,211 [myid:127.0.0.1:11263] - WARN [main-SendThread(127.0.0.1:11263):ClientCnxn$SendThread@1235] - Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.Net.connect0(Native Method)
at sun.nio.ch.Net.connect(Net.java:465)
at sun.nio.ch.Net.connect(Net.java:457)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670)
at org.apache.zookeeper.ClientCnxnSocketNIO.registerAndConnect(ClientCnxnSocketNIO.java:275)
at org.apache.zookeeper.ClientCnxnSocketNIO.connect(ClientCnxnSocketNIO.java:285)
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1098)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1133)
2016-07-28 09:30:23,231 [myid:127.0.0.1:11303] - INFO [main-SendThread(127.0.0.1:11303):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:11303. Will not attempt to authenticate using SASL (unknown error)
2016-07-28 09:30:23,231 [myid:127.0.0.1:11276] - INFO [main-SendThread(127.0.0.1:11276):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:11276. Will not attempt to authenticate using SASL (unknown error)
2016-07-28 09:30:23,231 [myid:127.0.0.1:11303] - ERROR [main-SendThread(127.0.0.1:11303):ClientCnxnSocketNIO@287] - Unable to open socket to 127.0.0.1/127.0.0.1:11303
2016-07-28 09:30:23,231 [myid:127.0.0.1:11303] - WARN [main-SendThread(127.0.0.1:11303):ClientCnxn$SendThread@1235] - Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.Net.connect0(Native Method)
at sun.nio.ch.Net.connect(Net.java:465)
at sun.nio.ch.Net.connect(Net.java:457)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670)
at org.apache.zookeeper.ClientCnxnSocketNIO.registerAndConnect(ClientCnxnSocketNIO.java:275)
at org.apache.zookeeper.ClientCnxnSocketNIO.connect(ClientCnxnSocketNIO.java:285)
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1098)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1133)
2016-07-28 09:30:23,231 [myid:127.0.0.1:11276] - ERROR [main-SendThread(127.0.0.1:11276):ClientCnxnSocketNIO@287] - Unable to open socket to 127.0.0.1/127.0.0.1:11276
2016-07-28 09:30:23,232 [myid:127.0.0.1:11276] - WARN [main-SendThread(127.0.0.1:11276):ClientCnxn$SendThread@1235] - Session 0x322add08d0d0000 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.Net.connect0(Native Method)
at sun.nio.ch.Net.connect(Net.java:465)
at sun.nio.ch.Net.connect(Net.java:457)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670)
at org.apache.zookeeper.ClientCnxnSocketNIO.registerAndConnect(ClientCnxnSocketNIO.java:275)
at org.apache.zookeeper.ClientCnxnSocketNIO.connect(ClientCnxnSocketNIO.java:285)
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1098)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1133)
2016-07-28 09:30:23,341 [myid:127.0.0.1:11257] - INFO [main-SendThread(127.0.0.1:11257):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:11257. Will not attempt to authenticate using SASL (unknown error)
2016-07-28 09:30:23,341 [myid:127.0.0.1:11257] - ERROR [main-SendThread(127.0.0.1:11257):ClientCnxnSocketNIO@287] - Unable to open socket to 127.0.0.1/127.0.0.1:11257
2016-07-28 09:30:23,341 [myid:127.0.0.1:11257] - WARN [main-SendThread(127.0.0.1:11257):ClientCnxn$SendThread@1235] - Session 0x122adcf0ae70000 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.Net.connect0(Native Method)
at sun.nio.ch.Net.connect(Net.java:465)
at sun.nio.ch.Net.connect(Net.java:457)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670)
at org.apache.zookeeper.ClientCnxnSocketNIO.registerAndConnect(ClientCnxnSocketNIO.java:275)
at org.apache.zookeeper.ClientCnxnSocketNIO.connect(ClientCnxnSocketNIO.java:285)
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1098)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1133)
2016-07-28 09:30:23,351 [myid:] - INFO [QuorumPeer[myid=4](plain=/127.0.0.1:11380)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id4,name1=replica.4,name2=Follower]
2016-07-28 09:30:23,351 [myid:] - INFO [QuorumPeer[myid=4](plain=/127.0.0.1:11380)(secure=disabled):Follower@198] - shutdown called
java.lang.Exception: shutdown Follower
at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:198)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1115)
2016-07-28 09:30:23,351 [myid:] - WARN [QuorumPeer[myid=4](plain=/127.0.0.1:11380)(secure=disabled):QuorumPeer@1158] - PeerState set to LOOKING
2016-07-28 09:30:23,351 [myid:] - WARN [QuorumPeer[myid=4](plain=/127.0.0.1:11380)(secure=disabled):QuorumPeer@1140] - QuorumPeer main thread exited
2016-07-28 09:30:23,351 [myid:] - INFO [QuorumPeer[myid=4](plain=/127.0.0.1:11380)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id4]
2016-07-28 09:30:23,352 [myid:] - INFO [QuorumPeer[myid=4](plain=/127.0.0.1:11380)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id4,name1=replica.4]
2016-07-28 09:30:23,352 [myid:] - INFO [QuorumPeer[myid=4](plain=/127.0.0.1:11380)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id4,name1=replica.1]
2016-07-28 09:30:23,352 [myid:] - INFO [QuorumPeer[myid=4](plain=/127.0.0.1:11380)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id4,name1=replica.2]
2016-07-28 09:30:23,352 [myid:] - INFO [QuorumPeer[myid=4](plain=/127.0.0.1:11380)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id4,name1=replica.3]
2016-07-28 09:30:23,352 [myid:] - INFO [QuorumPeer[myid=4](plain=/127.0.0.1:11380)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id4,name1=replica.5]
2016-07-28 09:30:23,352 [myid:] - INFO [main:QuorumUtil@250] - Shutting down quorum peer QuorumPeer[myid=5](plain=/127.0.0.1:11383)(secure=disabled)
2016-07-28 09:30:23,352 [myid:] - INFO [main:Leader@617] - Shutting down
2016-07-28 09:30:23,352 [myid:] - INFO [main:Leader@623] - Shutdown called
java.lang.Exception: shutdown Leader! reason: quorum Peer shutdown
at org.apache.zookeeper.server.quorum.Leader.shutdown(Leader.java:623)
at org.apache.zookeeper.server.quorum.QuorumPeer.shutdown(QuorumPeer.java:1181)
at org.apache.zookeeper.test.QuorumUtil.shutdown(QuorumUtil.java:251)
at org.apache.zookeeper.test.QuorumUtil.shutdownAll(QuorumUtil.java:238)
at org.apache.zookeeper.test.QuorumUtil.tearDown(QuorumUtil.java:306)
at org.apache.zookeeper.test.ReconfigTest.tearDown(ReconfigTest.java:64)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33)
at org.junit.rules.TestWatchman$1.evaluate(TestWatchman.java:53)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:38)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:518)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1052)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:906)
2016-07-28 09:30:23,353 [myid:] - INFO [main:ZooKeeperServer@498] - shutting down
2016-07-28 09:30:23,353 [myid:] - INFO [main:SessionTrackerImpl@232] - Shutting down
2016-07-28 09:30:23,353 [myid:] - INFO [main:LeaderRequestProcessor@77] - Shutting down
2016-07-28 09:30:23,353 [myid:] - INFO [main:PrepRequestProcessor@965] - Shutting down
2016-07-28 09:30:23,353 [myid:] - INFO [LearnerCnxAcceptor-/127.0.0.1:11384:Leader$LearnerCnxAcceptor@373] - exception while shutting down acceptor: java.net.SocketException: Socket closed
2016-07-28 09:30:23,353 [myid:] - INFO [ProcessThread(sid:5 cport:-1)::PrepRequestProcessor@154] - PrepRequestProcessor exited loop!
2016-07-28 09:30:23,353 [myid:] - INFO [main:ProposalRequestProcessor@88] - Shutting down
2016-07-28 09:30:23,354 [myid:] - INFO [main:CommitProcessor@414] - Shutting down
2016-07-28 09:30:23,354 [myid:] - INFO [CommitProcessor:5:CommitProcessor@299] - CommitProcessor exited loop!
2016-07-28 09:30:23,354 [myid:] - INFO [main:Leader$ToBeAppliedRequestProcessor@918] - Shutting down
2016-07-28 09:30:23,354 [myid:] - INFO [main:FinalRequestProcessor@479] - shutdown of request processor complete
2016-07-28 09:30:23,354 [myid:] - INFO [main:SyncRequestProcessor@191] - Shutting down
2016-07-28 09:30:23,354 [myid:] - INFO [SyncThread:5:SyncRequestProcessor@169] - SyncRequestProcessor exited!
2016-07-28 09:30:23,355 [myid:] - INFO [main:MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id5,name1=replica.5,name2=Leader,name3=InMemoryDataTree]
2016-07-28 09:30:23,355 [myid:] - WARN [LearnerHandler-/127.0.0.1:51992:LearnerHandler@619] - ******* GOODBYE /127.0.0.1:51992 ********
2016-07-28 09:30:23,355 [myid:] - WARN [LearnerHandler-/127.0.0.1:51977:LearnerHandler@619] - ******* GOODBYE /127.0.0.1:51977 ********
2016-07-28 09:30:23,355 [myid:] - WARN [LearnerHandler-/127.0.0.1:51992:LearnerHandler@903] - Ignoring unexpected exception
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1219)
at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:340)
at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:338)
at org.apache.zookeeper.server.quorum.LearnerHandler.shutdown(LearnerHandler.java:901)
at org.apache.zookeeper.server.quorum.LearnerHandler.run(LearnerHandler.java:622)
2016-07-28 09:30:23,355 [myid:] - WARN [LearnerHandler-/127.0.0.1:51976:LearnerHandler@619] - ******* GOODBYE /127.0.0.1:51976 ********
2016-07-28 09:30:23,355 [myid:] - WARN [LearnerHandler-/127.0.0.1:51975:LearnerHandler@619] - ******* GOODBYE /127.0.0.1:51975 ********
2016-07-28 09:30:23,356 [myid:] - INFO [ConnnectionExpirer:NIOServerCnxnFactory$ConnectionExpirerThread@583] - ConnnectionExpirerThread interrupted
2016-07-28 09:30:23,356 [myid:] - INFO [NIOServerCxnFactory.SelectorThread-0:NIOServerCnxnFactory$SelectorThread@420] - selector thread exitted run method
2016-07-28 09:30:23,355 [myid:] - WARN [LearnerHandler-/127.0.0.1:51977:LearnerHandler@903] - Ignoring unexpected exception
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1219)
at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:340)
at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:338)
at org.apache.zookeeper.server.quorum.LearnerHandler.shutdown(LearnerHandler.java:901)
at org.apache.zookeeper.server.quorum.LearnerHandler.run(LearnerHandler.java:622)
2016-07-28 09:30:23,356 [myid:] - INFO [NIOServerCxnFactory.AcceptThread:/127.0.0.1:11383:NIOServerCnxnFactory$AcceptThread@219] - accept thread exitted run method
2016-07-28 09:30:23,356 [myid:] - INFO [NIOServerCxnFactory.SelectorThread-1:NIOServerCnxnFactory$SelectorThread@420] - selector thread exitted run method
2016-07-28 09:30:23,356 [myid:] - WARN [LearnerHandler-/127.0.0.1:51975:LearnerHandler@903] - Ignoring unexpected exception
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1219)
at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:340)
at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:338)
at org.apache.zookeeper.server.quorum.LearnerHandler.shutdown(LearnerHandler.java:901)
at org.apache.zookeeper.server.quorum.LearnerHandler.run(LearnerHandler.java:622)
2016-07-28 09:30:23,356 [myid:] - WARN [LearnerHandler-/127.0.0.1:51976:LearnerHandler@903] - Ignoring unexpected exception
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1219)
at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:340)
at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:338)
at org.apache.zookeeper.server.quorum.LearnerHandler.shutdown(LearnerHandler.java:901)
at org.apache.zookeeper.server.quorum.LearnerHandler.run(LearnerHandler.java:622)
2016-07-28 09:30:23,357 [myid:] - INFO [/127.0.0.1:11385:QuorumCnxManager$Listener@661] - Leaving listener
2016-07-28 09:30:23,357 [myid:] - INFO [QuorumPeer[myid=5](plain=/127.0.0.1:11383)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id5,name1=replica.5,name2=Leader]
2016-07-28 09:30:23,358 [myid:] - INFO [main:QuorumUtil@254] - Shutting down leader election QuorumPeer[myid=5](plain=/127.0.0.1:11383)(secure=disabled)
2016-07-28 09:30:23,358 [myid:] - WARN [QuorumPeer[myid=5](plain=/127.0.0.1:11383)(secure=disabled):QuorumPeer@1127] - Unexpected exception
java.lang.InterruptedException
at java.lang.Object.wait(Native Method)
at org.apache.zookeeper.server.quorum.Leader.lead(Leader.java:561)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1124)
2016-07-28 09:30:23,358 [myid:] - INFO [QuorumPeer[myid=5](plain=/127.0.0.1:11383)(secure=disabled):Leader@617] - Shutting down
2016-07-28 09:30:23,358 [myid:] - WARN [QuorumPeer[myid=5](plain=/127.0.0.1:11383)(secure=disabled):QuorumPeer@1158] - PeerState set to LOOKING
2016-07-28 09:30:23,358 [myid:] - WARN [QuorumPeer[myid=5](plain=/127.0.0.1:11383)(secure=disabled):QuorumPeer@1140] - QuorumPeer main thread exited
2016-07-28 09:30:23,358 [myid:] - INFO [main:QuorumUtil@259] - Waiting for QuorumPeer[myid=5](plain=/127.0.0.1:11383)(secure=disabled) to exit thread
2016-07-28 09:30:23,358 [myid:] - INFO [QuorumPeer[myid=5](plain=/127.0.0.1:11383)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id5]
2016-07-28 09:30:23,359 [myid:] - INFO [QuorumPeer[myid=5](plain=/127.0.0.1:11383)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id5,name1=replica.5]
2016-07-28 09:30:23,359 [myid:] - INFO [QuorumPeer[myid=5](plain=/127.0.0.1:11383)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id5,name1=replica.1]
2016-07-28 09:30:23,359 [myid:] - INFO [QuorumPeer[myid=5](plain=/127.0.0.1:11383)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id5,name1=replica.2]
2016-07-28 09:30:23,359 [myid:] - INFO [QuorumPeer[myid=5](plain=/127.0.0.1:11383)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id5,name1=replica.3]
2016-07-28 09:30:23,359 [myid:] - INFO [main:FourLetterWordMain@85] - connecting to 127.0.0.1 11371
2016-07-28 09:30:23,359 [myid:] - INFO [main:QuorumUtil@243] - 127.0.0.1:11371 is no longer accepting client connections
2016-07-28 09:30:23,359 [myid:] - INFO [main:FourLetterWordMain@85] - connecting to 127.0.0.1 11374
2016-07-28 09:30:23,360 [myid:] - INFO [main:QuorumUtil@243] - 127.0.0.1:11374 is no longer accepting client connections
2016-07-28 09:30:23,360 [myid:] - INFO [main:FourLetterWordMain@85] - connecting to 127.0.0.1 11377
2016-07-28 09:30:23,360 [myid:] - INFO [main:QuorumUtil@243] - 127.0.0.1:11377 is no longer accepting client connections
2016-07-28 09:30:23,360 [myid:] - INFO [main:FourLetterWordMain@85] - connecting to 127.0.0.1 11380
2016-07-28 09:30:23,360 [myid:] - INFO [main:QuorumUtil@243] - 127.0.0.1:11380 is no longer accepting client connections
2016-07-28 09:30:23,360 [myid:] - INFO [main:FourLetterWordMain@85] - connecting to 127.0.0.1 11383
2016-07-28 09:30:23,360 [myid:] - INFO [main:QuorumUtil@243] - 127.0.0.1:11383 is no longer accepting client connections
2016-07-28 09:30:23,361 [myid:] - INFO [main:ZKTestCase$1@65] - SUCCEEDED testRemoveOneAsynchronous
2016-07-28 09:30:23,361 [myid:] - INFO [main:ZKTestCase$1@60] - FINISHED testRemoveOneAsynchronous
{noformat}
flaky, flaky-test 9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 34 weeks ago 0|i31mu7:
ZooKeeper ZOOKEEPER-2492

gethostname return error before Win32WSAStartup on windows.

Bug Patch Available Trivial Unresolved spooky000 spooky000 spooky000 27/Jul/16 22:09   05/Feb/20 07:12   3.5.2 3.7.0, 3.5.8     0 2   windows gethostname return error before Win32WSAStartup on windows.

in log_env function.
gethostname(buf, sizeof(buf));
LOG_INFO(LOGCALLBACK(zh), "Client environment:host.name=%s", buf);

buf will be uninitialized buffer.
9223372036854775807 No Perforce job exists for this issue. 3 9223372036854775807
1 year, 45 weeks ago 0|i31lr3:
ZooKeeper ZOOKEEPER-2491

C client build error in vs 2015

Bug Resolved Minor Duplicate Andrew Schwartzmeyer spooky000 spooky000 27/Jul/16 21:59   19/Dec/19 18:01 27/Jul/17 18:03 3.5.2 3.5.4 c client   0 4   windows vs 2015 Visual Studio 2015 supports snprintf.
#define snprintf _snprintf throw error.
9223372036854775807 No Perforce job exists for this issue. 3 9223372036854775807
2 years, 34 weeks ago 0|i31lqv:
ZooKeeper ZOOKEEPER-2490

infinitely connect on windows

Bug Patch Available Major Unresolved spooky000 spooky000 spooky000 27/Jul/16 02:17   28/Feb/20 16:56   3.5.2 3.7.0, 3.5.8 c client   0 3 0 600   Windows in addrvec_contains function
this memcmp always return false on windows release build.

for (i = 0; i < avec->count; i++)
{
if(memcmp(&avec->data[i], addr, INET_ADDRSTRLEN) == 0)
return 1;
}

because..
#define INET_ADDRSTRLEN 16 on linux.
#define INET_ADDRSTRLEN 22 on windows.




100% 100% 600 0 pull-request-available 9223372036854775807 No Perforce job exists for this issue. 3 9223372036854775807
2 years, 36 weeks ago 0|i31jvz:
ZooKeeper ZOOKEEPER-2489

Upgrade Jetty dependency to a recent stable release version.

Improvement Closed Major Fixed Edward Ribeiro Michael Han Michael Han 26/Jul/16 16:54   17/May/17 23:44 24/Aug/16 20:25 3.4.8, 3.5.2 3.5.3, 3.6.0 server   0 4   Jetty was added as one of dependencies in ZOOKEEPER-1346 in 2011, and have not been updated since then. The version we are using in trunk is 6.1.26 which was released in 2010. We should consider upgrade Jetty to a recent stable release version (probably one of the 9.x.).

Note: this JIRA issue is recreated from https://issues.apache.org/jira/browse/ZOOKEEPER-2427, which was deleted a couple of weeks ago as part of Apache Infrastructure spam fighting effort.
9223372036854775807 No Perforce job exists for this issue. 3 9223372036854775807
3 years, 30 weeks ago
Reviewed
0|i31ja7:
ZooKeeper ZOOKEEPER-2488

Unsynchronized access to shuttingDownLE in QuorumPeer

Bug Open Major Unresolved gaoshu Michael Han Michael Han 25/Jul/16 19:05   05/Feb/20 07:16   3.5.2 3.7.0, 3.5.8 server   0 5 0 2400   Access to shuttingDownLE in QuorumPeer is not synchronized here:

https://github.com/apache/zookeeper/blob/3c37184e83a3e68b73544cebccf9388eea26f523/src/java/main/org/apache/zookeeper/server/quorum/QuorumPeer.java#L1066
https://github.com/apache/zookeeper/blob/3c37184e83a3e68b73544cebccf9388eea26f523/src/java/main/org/

The access should be synchronized as the same variable might be accessed
in QuormPeer::restartLeaderElection, which is synchronized.
100% 100% 2400 0 pull-request-available 9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
1 year, 16 weeks, 3 days ago 0|i31hgv:
ZooKeeper ZOOKEEPER-2487

Test failed because of 'Forked Java VM exited abnormally.'

Test Open Major Unresolved Michael Han Michael Han Michael Han 25/Jul/16 13:06   20/Nov/19 05:41   3.4.8, 3.4.11   tests   0 2   Sometimes tests fail with error message like this:

Error Message:
Forked Java VM exited abnormally. Please note the time in the report does not reflect the time until the VM exit.

Stack Trace:
junit.framework.AssertionFailedError: Forked Java VM exited abnormally. Please note the time in the report does not reflect the time until the VM exit.

Examples:
https://builds.apache.org/job/ZooKeeper-trunk-solaris/1239/
https://builds.apache.org/job/ZooKeeper_branch34_openjdk7/1147/
https://builds.apache.org/job/ZooKeeper_branch34_openjdk7/1129/

The failure happen on all platforms (jdk7/8/solaris) of branch 3.4; branch 3.5 looks ok in general.
9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
17 weeks, 1 day ago 0|i31gyv:
ZooKeeper ZOOKEEPER-2486

ZOOKEEPER-3170 Flaky Test: org.apache.zookeeper.test.QuorumZxidSyncTest.testBehindLeader

Sub-task Resolved Major Cannot Reproduce Andor Molnar Michael Han Michael Han 21/Jul/16 18:44   25/Oct/18 10:55 25/Oct/18 10:55 3.4.8 3.5.2 tests   0 2   From https://builds.apache.org/job/ZooKeeper_branch34_jdk7/1156

{noformat}
Error Message

waiting for server up
Stacktrace

junit.framework.AssertionFailedError: waiting for server up
at org.apache.zookeeper.test.QuorumBase.startServers(QuorumBase.java:183)
at org.apache.zookeeper.test.QuorumBase.startServers(QuorumBase.java:113)
at org.apache.zookeeper.test.QuorumZxidSyncTest.testBehindLeader(QuorumZxidSyncTest.java:67)
at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:55)
Standard Output

2016-07-21 08:11:45,722 [myid:] - INFO [main:PortAssignment@32] - assigning port 11221
2016-07-21 08:11:45,729 [myid:] - INFO [main:ZKTestCase$1@50] - STARTING testBehindLeader
2016-07-21 08:11:45,729 [myid:] - INFO [main:QuorumBase@69] - QuorumBase.setup null
2016-07-21 08:11:45,926 [myid:] - INFO [main:PortAssignment@32] - assigning port 11222
2016-07-21 08:11:45,926 [myid:] - INFO [main:PortAssignment@32] - assigning port 11223
2016-07-21 08:11:45,927 [myid:] - INFO [main:PortAssignment@32] - assigning port 11224
2016-07-21 08:11:45,927 [myid:] - INFO [main:PortAssignment@32] - assigning port 11225
2016-07-21 08:11:45,927 [myid:] - INFO [main:PortAssignment@32] - assigning port 11226
2016-07-21 08:11:45,928 [myid:] - INFO [main:PortAssignment@32] - assigning port 11227
2016-07-21 08:11:45,928 [myid:] - INFO [main:PortAssignment@32] - assigning port 11228
2016-07-21 08:11:45,928 [myid:] - INFO [main:PortAssignment@32] - assigning port 11229
2016-07-21 08:11:45,928 [myid:] - INFO [main:PortAssignment@32] - assigning port 11230
2016-07-21 08:11:45,929 [myid:] - INFO [main:PortAssignment@32] - assigning port 11231
2016-07-21 08:11:45,929 [myid:] - INFO [main:QuorumBase@93] - Ports are: 127.0.0.1:11222,127.0.0.1:11223,127.0.0.1:11224,127.0.0.1:11225,127.0.0.1:11226
2016-07-21 08:11:45,946 [myid:] - INFO [main:QuorumPeer$QuorumServer@149] - Resolved hostname: 127.0.0.1 to address: /127.0.0.1
2016-07-21 08:11:45,946 [myid:] - INFO [main:QuorumPeer$QuorumServer@149] - Resolved hostname: 127.0.0.1 to address: /127.0.0.1
2016-07-21 08:11:45,948 [myid:] - INFO [main:QuorumPeer$QuorumServer@149] - Resolved hostname: 127.0.0.1 to address: /127.0.0.1
2016-07-21 08:11:45,948 [myid:] - INFO [main:QuorumPeer$QuorumServer@149] - Resolved hostname: 127.0.0.1 to address: /127.0.0.1
2016-07-21 08:11:45,949 [myid:] - INFO [main:QuorumPeer$QuorumServer@149] - Resolved hostname: 127.0.0.1 to address: /127.0.0.1
2016-07-21 08:11:45,949 [myid:] - INFO [main:QuorumBase@142] - creating QuorumPeer 1 port 11222
2016-07-21 08:11:45,961 [myid:] - INFO [main:NIOServerCnxnFactory@89] - binding to port 0.0.0.0/0.0.0.0:11222
2016-07-21 08:11:45,979 [myid:] - INFO [main:QuorumBase@145] - creating QuorumPeer 2 port 11223
2016-07-21 08:11:45,979 [myid:] - INFO [main:NIOServerCnxnFactory@89] - binding to port 0.0.0.0/0.0.0.0:11223
2016-07-21 08:11:45,980 [myid:] - INFO [main:QuorumBase@148] - creating QuorumPeer 3 port 11224
2016-07-21 08:11:45,980 [myid:] - INFO [main:NIOServerCnxnFactory@89] - binding to port 0.0.0.0/0.0.0.0:11224
2016-07-21 08:11:45,981 [myid:] - INFO [main:QuorumBase@151] - creating QuorumPeer 4 port 11225
2016-07-21 08:11:45,981 [myid:] - INFO [main:NIOServerCnxnFactory@89] - binding to port 0.0.0.0/0.0.0.0:11225
2016-07-21 08:11:45,981 [myid:] - INFO [main:QuorumBase@154] - creating QuorumPeer 5 port 11226
2016-07-21 08:11:45,982 [myid:] - INFO [main:NIOServerCnxnFactory@89] - binding to port 0.0.0.0/0.0.0.0:11226
2016-07-21 08:11:45,982 [myid:] - INFO [main:QuorumBase@163] - QuorumPeer 1 voting view: {1=org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@47628981, 2=org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@163198c4, 3=org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@1224773e, 4=org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@181090c0, 5=org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@678f3997}
2016-07-21 08:11:45,983 [myid:] - INFO [main:QuorumBase@164] - QuorumPeer 2 voting view: {1=org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@47628981, 2=org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@163198c4, 3=org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@1224773e, 4=org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@181090c0, 5=org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@678f3997}
2016-07-21 08:11:45,983 [myid:] - INFO [main:QuorumBase@165] - QuorumPeer 3 voting view: {1=org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@47628981, 2=org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@163198c4, 3=org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@1224773e, 4=org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@181090c0, 5=org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@678f3997}
2016-07-21 08:11:45,983 [myid:] - INFO [main:QuorumBase@166] - QuorumPeer 4 voting view: {1=org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@47628981, 2=org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@163198c4, 3=org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@1224773e, 4=org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@181090c0, 5=org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@678f3997}
2016-07-21 08:11:45,983 [myid:] - INFO [main:QuorumBase@167] - QuorumPeer 5 voting view: {1=org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@47628981, 2=org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@163198c4, 3=org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@1224773e, 4=org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@181090c0, 5=org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@678f3997}
2016-07-21 08:11:45,983 [myid:] - INFO [main:QuorumBase@169] - start QuorumPeer 1
2016-07-21 08:11:45,988 [myid:] - INFO [main:QuorumPeer@533] - currentEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
2016-07-21 08:11:45,990 [myid:] - INFO [main:QuorumPeer@548] - acceptedEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
2016-07-21 08:11:45,996 [myid:] - INFO [ListenerThread:QuorumCnxManager$Listener@534] - My election bind port: /127.0.0.1:12227
2016-07-21 08:11:46,001 [myid:] - INFO [main:QuorumBase@171] - start QuorumPeer 2
2016-07-21 08:11:46,002 [myid:] - INFO [main:QuorumPeer@533] - currentEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
2016-07-21 08:11:46,003 [myid:] - INFO [main:QuorumPeer@548] - acceptedEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
2016-07-21 08:11:46,004 [myid:] - INFO [ListenerThread:QuorumCnxManager$Listener@534] - My election bind port: /127.0.0.1:12228
2016-07-21 08:11:46,006 [myid:] - INFO [main:QuorumBase@173] - start QuorumPeer 3
2016-07-21 08:11:46,006 [myid:] - INFO [main:QuorumPeer@533] - currentEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
2016-07-21 08:11:46,007 [myid:] - INFO [main:QuorumPeer@548] - acceptedEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
2016-07-21 08:11:46,007 [myid:] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11222:QuorumPeer@774] - LOOKING
2016-07-21 08:11:46,008 [myid:] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11223:QuorumPeer@774] - LOOKING
2016-07-21 08:11:46,009 [myid:] - INFO [ListenerThread:QuorumCnxManager$Listener@534] - My election bind port: /127.0.0.1:12229
2016-07-21 08:11:46,009 [myid:] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11222:FastLeaderElection@818] - New election. My id = 1, proposed zxid=0x0
2016-07-21 08:11:46,009 [myid:] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11223:FastLeaderElection@818] - New election. My id = 2, proposed zxid=0x0
2016-07-21 08:11:46,009 [myid:] - INFO [main:QuorumBase@175] - start QuorumPeer 4
2016-07-21 08:11:46,010 [myid:] - INFO [main:QuorumPeer@533] - currentEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
2016-07-21 08:11:46,011 [myid:] - INFO [main:QuorumPeer@548] - acceptedEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
2016-07-21 08:11:46,011 [myid:] - INFO [QuorumPeer[myid=3]/0:0:0:0:0:0:0:0:11224:QuorumPeer@774] - LOOKING
2016-07-21 08:11:46,011 [myid:] - INFO [/127.0.0.1:12228:QuorumCnxManager$Listener@541] - Received connection request /127.0.0.1:38924
2016-07-21 08:11:46,011 [myid:] - INFO [QuorumPeer[myid=3]/0:0:0:0:0:0:0:0:11224:FastLeaderElection@818] - New election. My id = 3, proposed zxid=0x0
2016-07-21 08:11:46,011 [myid:] - INFO [/127.0.0.1:12227:QuorumCnxManager$Listener@541] - Received connection request /127.0.0.1:43415
2016-07-21 08:11:46,011 [myid:] - INFO [WorkerSender[myid=1]:QuorumCnxManager@199] - Have smaller server identifier, so dropping the connection: (2, 1)
2016-07-21 08:11:46,011 [myid:] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message format version), 1 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,013 [myid:] - INFO [WorkerSender[myid=1]:QuorumCnxManager@199] - Have smaller server identifier, so dropping the connection: (3, 1)
2016-07-21 08:11:46,014 [myid:] - INFO [/127.0.0.1:12229:QuorumCnxManager$Listener@541] - Received connection request /127.0.0.1:51548
2016-07-21 08:11:46,014 [myid:] - INFO [main:QuorumBase@177] - start QuorumPeer 5
2016-07-21 08:11:46,015 [myid:] - INFO [ListenerThread:QuorumCnxManager$Listener@534] - My election bind port: /127.0.0.1:12230
2016-07-21 08:11:46,015 [myid:] - INFO [main:QuorumPeer@533] - currentEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
2016-07-21 08:11:46,015 [myid:] - WARN [WorkerSender[myid=1]:QuorumCnxManager@400] - Cannot open channel to 4 at election address /127.0.0.1:12230
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:381)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:354)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:452)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:433)
at java.lang.Thread.run(Thread.java:745)
2016-07-21 08:11:46,018 [myid:] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,018 [myid:] - INFO [/127.0.0.1:12229:QuorumCnxManager$Listener@541] - Received connection request /127.0.0.1:51551
2016-07-21 08:11:46,016 [myid:] - INFO [/127.0.0.1:12227:QuorumCnxManager$Listener@541] - Received connection request /127.0.0.1:43417
2016-07-21 08:11:46,016 [myid:] - INFO [QuorumPeer[myid=4]/0:0:0:0:0:0:0:0:11225:QuorumPeer@774] - LOOKING
2016-07-21 08:11:46,016 [myid:] - INFO [main:QuorumPeer@548] - acceptedEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
2016-07-21 08:11:46,019 [myid:] - INFO [QuorumPeer[myid=4]/0:0:0:0:0:0:0:0:11225:FastLeaderElection@818] - New election. My id = 4, proposed zxid=0x0
2016-07-21 08:11:46,019 [myid:] - INFO [/127.0.0.1:12230:QuorumCnxManager$Listener@541] - Received connection request /127.0.0.1:46102
2016-07-21 08:11:46,018 [myid:] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@600] - Notification: 1 (message format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,018 [myid:] - INFO [WorkerSender[myid=2]:QuorumCnxManager@199] - Have smaller server identifier, so dropping the connection: (3, 2)
2016-07-21 08:11:46,018 [myid:] - INFO [WorkerSender[myid=1]:QuorumPeer$QuorumServer@149] - Resolved hostname: 127.0.0.1 to address: /127.0.0.1
2016-07-21 08:11:46,018 [myid:] - INFO [/127.0.0.1:12228:QuorumCnxManager$Listener@541] - Received connection request /127.0.0.1:38928
2016-07-21 08:11:46,021 [myid:] - INFO [WorkerSender[myid=2]:QuorumCnxManager@199] - Have smaller server identifier, so dropping the connection: (4, 2)
2016-07-21 08:11:46,021 [myid:] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@600] - Notification: 1 (message format version), 1 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,021 [myid:] - INFO [/127.0.0.1:12229:QuorumCnxManager$Listener@541] - Received connection request /127.0.0.1:51553
2016-07-21 08:11:46,020 [myid:] - INFO [WorkerReceiver[myid=3]:FastLeaderElection@600] - Notification: 1 (message format version), 3 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 3 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,019 [myid:] - INFO [WorkerSender[myid=3]:QuorumCnxManager@199] - Have smaller server identifier, so dropping the connection: (4, 3)
2016-07-21 08:11:46,022 [myid:] - INFO [WorkerReceiver[myid=3]:FastLeaderElection@600] - Notification: 1 (message format version), 1 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,021 [myid:] - WARN [WorkerSender[myid=2]:QuorumCnxManager@400] - Cannot open channel to 5 at election address /127.0.0.1:12231
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:381)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:354)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:452)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:433)
at java.lang.Thread.run(Thread.java:745)
2016-07-21 08:11:46,021 [myid:] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message format version), 3 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 3 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,021 [myid:] - WARN [WorkerSender[myid=1]:QuorumCnxManager@400] - Cannot open channel to 5 at election address /127.0.0.1:12231
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:381)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:354)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:452)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:433)
at java.lang.Thread.run(Thread.java:745)
2016-07-21 08:11:46,023 [myid:] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@600] - Notification: 1 (message format version), 3 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 3 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,023 [myid:] - INFO [WorkerReceiver[myid=3]:FastLeaderElection@600] - Notification: 1 (message format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,023 [myid:] - INFO [WorkerSender[myid=2]:QuorumPeer$QuorumServer@149] - Resolved hostname: 127.0.0.1 to address: /127.0.0.1
2016-07-21 08:11:46,024 [myid:] - INFO [/127.0.0.1:12230:QuorumCnxManager$Listener@541] - Received connection request /127.0.0.1:46104
2016-07-21 08:11:46,022 [myid:] - INFO [ListenerThread:QuorumCnxManager$Listener@534] - My election bind port: /127.0.0.1:12231
2016-07-21 08:11:46,022 [myid:] - WARN [WorkerSender[myid=3]:QuorumCnxManager@400] - Cannot open channel to 5 at election address /127.0.0.1:12231
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:381)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:354)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:452)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:433)
at java.lang.Thread.run(Thread.java:745)
2016-07-21 08:11:46,025 [myid:] - INFO [WorkerReceiver[myid=3]:FastLeaderElection@600] - Notification: 1 (message format version), 3 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,025 [myid:] - INFO [WorkerReceiver[myid=4]:FastLeaderElection@600] - Notification: 1 (message format version), 3 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 3 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,025 [myid:] - INFO [main:QuorumBase@179] - started QuorumPeer 5
2016-07-21 08:11:46,026 [myid:] - INFO [main:QuorumBase@181] - Checking ports 127.0.0.1:11222,127.0.0.1:11223,127.0.0.1:11224,127.0.0.1:11225,127.0.0.1:11226
2016-07-21 08:11:46,025 [myid:] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message format version), 3 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,025 [myid:] - INFO [/127.0.0.1:12227:QuorumCnxManager$Listener@541] - Received connection request /127.0.0.1:43428
2016-07-21 08:11:46,027 [myid:] - INFO [/127.0.0.1:12230:QuorumCnxManager$Listener@541] - Received connection request /127.0.0.1:46109
2016-07-21 08:11:46,025 [myid:] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@600] - Notification: 1 (message format version), 3 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,024 [myid:] - INFO [WorkerSender[myid=1]:QuorumPeer$QuorumServer@149] - Resolved hostname: 127.0.0.1 to address: /127.0.0.1
2016-07-21 08:11:46,027 [myid:] - INFO [WorkerSender[myid=4]:QuorumCnxManager@199] - Have smaller server identifier, so dropping the connection: (5, 4)
2016-07-21 08:11:46,027 [myid:] - INFO [WorkerReceiver[myid=4]:FastLeaderElection@600] - Notification: 1 (message format version), 4 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 4 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,026 [myid:] - INFO [/127.0.0.1:12228:QuorumCnxManager$Listener@541] - Received connection request /127.0.0.1:38938
2016-07-21 08:11:46,025 [myid:] - INFO [WorkerSender[myid=3]:QuorumPeer$QuorumServer@149] - Resolved hostname: 127.0.0.1 to address: /127.0.0.1
2016-07-21 08:11:46,025 [myid:] - INFO [WorkerSender[myid=2]:QuorumCnxManager@199] - Have smaller server identifier, so dropping the connection: (4, 2)
2016-07-21 08:11:46,028 [myid:] - INFO [main:FourLetterWordMain@62] - connecting to 127.0.0.1 11222
2016-07-21 08:11:46,028 [myid:] - WARN [RecvWorker:2:QuorumCnxManager$RecvWorker@810] - Connection broken for id 2, my id = 4, error =
java.net.SocketException: Socket closed
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at java.net.SocketInputStream.read(SocketInputStream.java:210)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:795)
2016-07-21 08:11:46,029 [myid:] - WARN [RecvWorker:2:QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker
2016-07-21 08:11:46,028 [myid:] - INFO [WorkerReceiver[myid=3]:FastLeaderElection@600] - Notification: 1 (message format version), 4 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 4 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,027 [myid:] - INFO [QuorumPeer[myid=5]/0:0:0:0:0:0:0:0:11226:QuorumPeer@774] - LOOKING
2016-07-21 08:11:46,027 [myid:] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,027 [myid:] - INFO [/127.0.0.1:12231:QuorumCnxManager$Listener@541] - Received connection request /127.0.0.1:52086
2016-07-21 08:11:46,030 [myid:] - WARN [RecvWorker:4:QuorumCnxManager$RecvWorker@810] - Connection broken for id 4, my id = 2, error =
java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:392)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:795)
2016-07-21 08:11:46,030 [myid:] - WARN [SendWorker:4:QuorumCnxManager$SendWorker@732] - Exception when using channel: for id 4 my id = 2 error = java.net.SocketException: Broken pipe
2016-07-21 08:11:46,029 [myid:] - INFO [WorkerReceiver[myid=3]:FastLeaderElection@600] - Notification: 1 (message format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,031 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11222:NIOServerCnxnFactory@192] - Accepted socket connection from /127.0.0.1:52363
2016-07-21 08:11:46,029 [myid:] - INFO [/127.0.0.1:12228:QuorumCnxManager$Listener@541] - Received connection request /127.0.0.1:38940
2016-07-21 08:11:46,029 [myid:] - INFO [WorkerSender[myid=1]:QuorumCnxManager@199] - Have smaller server identifier, so dropping the connection: (5, 1)
2016-07-21 08:11:46,029 [myid:] - INFO [WorkerReceiver[myid=4]:FastLeaderElection@600] - Notification: 1 (message format version), 1 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,032 [myid:] - INFO [WorkerReceiver[myid=4]:FastLeaderElection@600] - Notification: 1 (message format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,032 [myid:] - INFO [WorkerReceiver[myid=4]:FastLeaderElection@600] - Notification: 1 (message format version), 4 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 3 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,029 [myid:] - WARN [SendWorker:2:QuorumCnxManager$SendWorker@732] - Exception when using channel: for id 2 my id = 4 error = java.net.SocketException: Socket closed
2016-07-21 08:11:46,033 [myid:] - WARN [SendWorker:2:QuorumCnxManager$SendWorker@736] - Send worker leaving thread
2016-07-21 08:11:46,029 [myid:] - INFO [WorkerSender[myid=2]:QuorumCnxManager@199] - Have smaller server identifier, so dropping the connection: (5, 2)
2016-07-21 08:11:46,028 [myid:] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@600] - Notification: 1 (message format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,033 [myid:] - INFO [WorkerSender[myid=1]:QuorumCnxManager@199] - Have smaller server identifier, so dropping the connection: (5, 1)
2016-07-21 08:11:46,032 [myid:] - INFO [WorkerReceiver[myid=4]:FastLeaderElection@600] - Notification: 1 (message format version), 3 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,031 [myid:] - INFO [/127.0.0.1:12231:QuorumCnxManager$Listener@541] - Received connection request /127.0.0.1:52088
2016-07-21 08:11:46,031 [myid:] - INFO [WorkerReceiver[myid=3]:FastLeaderElection@600] - Notification: 1 (message format version), 4 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 3 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,031 [myid:] - INFO [/127.0.0.1:12230:QuorumCnxManager$Listener@541] - Received connection request /127.0.0.1:46117
2016-07-21 08:11:46,030 [myid:] - WARN [SendWorker:4:QuorumCnxManager$SendWorker@736] - Send worker leaving thread
2016-07-21 08:11:46,030 [myid:] - WARN [RecvWorker:4:QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker
2016-07-21 08:11:46,030 [myid:] - INFO [WorkerSender[myid=3]:QuorumCnxManager@199] - Have smaller server identifier, so dropping the connection: (5, 3)
2016-07-21 08:11:46,030 [myid:] - INFO [QuorumPeer[myid=5]/0:0:0:0:0:0:0:0:11226:FastLeaderElection@818] - New election. My id = 5, proposed zxid=0x0
2016-07-21 08:11:46,030 [myid:] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message format version), 4 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 4 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,034 [myid:] - INFO [/127.0.0.1:12228:QuorumCnxManager$Listener@541] - Received connection request /127.0.0.1:38947
2016-07-21 08:11:46,034 [myid:] - INFO [WorkerReceiver[myid=3]:FastLeaderElection@600] - Notification: 1 (message format version), 3 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,034 [myid:] - INFO [WorkerReceiver[myid=4]:FastLeaderElection@600] - Notification: 1 (message format version), 3 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,033 [myid:] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@600] - Notification: 1 (message format version), 4 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 3 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,035 [myid:] - INFO [/127.0.0.1:12231:QuorumCnxManager$Listener@541] - Received connection request /127.0.0.1:52089
2016-07-21 08:11:46,035 [myid:] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message format version), 4 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 3 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,036 [myid:] - INFO [WorkerReceiver[myid=4]:FastLeaderElection@600] - Notification: 1 (message format version), 4 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,036 [myid:] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@600] - Notification: 1 (message format version), 3 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,037 [myid:] - INFO [/127.0.0.1:12227:QuorumCnxManager$Listener@541] - Received connection request /127.0.0.1:43442
2016-07-21 08:11:46,036 [myid:] - INFO [WorkerReceiver[myid=5]:FastLeaderElection@600] - Notification: 1 (message format version), 4 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 4 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,036 [myid:] - INFO [WorkerReceiver[myid=3]:FastLeaderElection@600] - Notification: 1 (message format version), 4 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,037 [myid:] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@600] - Notification: 1 (message format version), 4 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 4 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,037 [myid:] - INFO [WorkerReceiver[myid=4]:FastLeaderElection@600] - Notification: 1 (message format version), 4 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,038 [myid:] - INFO [/127.0.0.1:12231:QuorumCnxManager$Listener@541] - Received connection request /127.0.0.1:52091
2016-07-21 08:11:46,037 [myid:] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message format version), 3 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,036 [myid:] - INFO [WorkerSender[myid=1]:QuorumCnxManager@199] - Have smaller server identifier, so dropping the connection: (5, 1)
2016-07-21 08:11:46,038 [myid:] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message format version), 4 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,038 [myid:] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message format version), 4 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,038 [myid:] - INFO [WorkerReceiver[myid=5]:FastLeaderElection@600] - Notification: 1 (message format version), 4 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,038 [myid:] - INFO [/127.0.0.1:12229:QuorumCnxManager$Listener@541] - Received connection request /127.0.0.1:51573
2016-07-21 08:11:46,038 [myid:] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@600] - Notification: 1 (message format version), 4 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,037 [myid:] - INFO [WorkerReceiver[myid=3]:FastLeaderElection@600] - Notification: 1 (message format version), 4 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,039 [myid:] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 5 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,039 [myid:] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@600] - Notification: 1 (message format version), 4 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,039 [myid:] - INFO [WorkerReceiver[myid=5]:FastLeaderElection@600] - Notification: 1 (message format version), 4 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,040 [myid:] - INFO [WorkerReceiver[myid=4]:FastLeaderElection@600] - Notification: 1 (message format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 5 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,039 [myid:] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,039 [myid:] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@600] - Notification: 1 (message format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 5 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,039 [myid:] - INFO [/127.0.0.1:12231:QuorumCnxManager$Listener@541] - Received connection request /127.0.0.1:52093
2016-07-21 08:11:46,040 [myid:] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@600] - Notification: 1 (message format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,040 [myid:] - INFO [WorkerReceiver[myid=3]:FastLeaderElection@600] - Notification: 1 (message format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,040 [myid:] - INFO [WorkerReceiver[myid=4]:FastLeaderElection@600] - Notification: 1 (message format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,040 [myid:] - INFO [WorkerReceiver[myid=5]:FastLeaderElection@600] - Notification: 1 (message format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 5 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,041 [myid:] - INFO [WorkerReceiver[myid=4]:FastLeaderElection@600] - Notification: 1 (message format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,041 [myid:] - INFO [/127.0.0.1:12227:QuorumCnxManager$Listener@541] - Received connection request /127.0.0.1:43444
2016-07-21 08:11:46,041 [myid:] - INFO [WorkerReceiver[myid=4]:FastLeaderElection@600] - Notification: 1 (message format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 4 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,041 [myid:] - WARN [RecvWorker:1:QuorumCnxManager$RecvWorker@810] - Connection broken for id 1, my id = 5, error =
java.net.SocketException: Socket closed
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at java.io.DataInputStream.readFully(DataInputStream.java:195)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:805)
2016-07-21 08:11:46,042 [myid:] - WARN [RecvWorker:1:QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker
2016-07-21 08:11:46,041 [myid:] - INFO [WorkerReceiver[myid=3]:FastLeaderElection@600] - Notification: 1 (message format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 4 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,041 [myid:] - WARN [SendWorker:1:QuorumCnxManager$SendWorker@727] - Interrupted while waiting for message on queue
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:879)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:65)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:715)
2016-07-21 08:11:46,041 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11222:NIOServerCnxn@827] - Processing stat command from /127.0.0.1:52363
2016-07-21 08:11:46,041 [myid:] - WARN [RecvWorker:5:QuorumCnxManager$RecvWorker@810] - Connection broken for id 5, my id = 1, error =
java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:392)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:795)
2016-07-21 08:11:46,043 [myid:] - WARN [RecvWorker:5:QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker
2016-07-21 08:11:46,041 [myid:] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@600] - Notification: 1 (message format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,041 [myid:] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,043 [myid:] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@600] - Notification: 1 (message format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 4 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,043 [myid:] - WARN [SendWorker:1:QuorumCnxManager$SendWorker@736] - Send worker leaving thread
2016-07-21 08:11:46,042 [myid:] - INFO [WorkerReceiver[myid=3]:FastLeaderElection@600] - Notification: 1 (message format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 5 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,042 [myid:] - WARN [SendWorker:5:QuorumCnxManager$SendWorker@727] - Interrupted while waiting for message on queue
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:879)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:65)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:715)
2016-07-21 08:11:46,044 [myid:] - WARN [SendWorker:5:QuorumCnxManager$SendWorker@736] - Send worker leaving thread
2016-07-21 08:11:46,042 [myid:] - INFO [WorkerReceiver[myid=4]:FastLeaderElection@600] - Notification: 1 (message format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 3 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,042 [myid:] - INFO [/127.0.0.1:12231:QuorumCnxManager$Listener@541] - Received connection request /127.0.0.1:52096
2016-07-21 08:11:46,041 [myid:] - INFO [WorkerReceiver[myid=5]:FastLeaderElection@600] - Notification: 1 (message format version), 4 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 3 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,045 [myid:] - WARN [SendWorker:1:QuorumCnxManager$SendWorker@727] - Interrupted while waiting for message on queue
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:879)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:65)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:715)
2016-07-21 08:11:46,045 [myid:] - WARN [SendWorker:1:QuorumCnxManager$SendWorker@736] - Send worker leaving thread
2016-07-21 08:11:46,045 [myid:] - INFO [Thread-2:NIOServerCnxn@1008] - Closed socket connection for client /127.0.0.1:52363 (no session established for client)
2016-07-21 08:11:46,044 [myid:] - INFO [WorkerReceiver[myid=3]:FastLeaderElection@600] - Notification: 1 (message format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,044 [myid:] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@600] - Notification: 1 (message format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 3 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,044 [myid:] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 4 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,046 [myid:] - INFO [WorkerReceiver[myid=3]:FastLeaderElection@600] - Notification: 1 (message format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 3 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,045 [myid:] - INFO [/127.0.0.1:12227:QuorumCnxManager$Listener@541] - Received connection request /127.0.0.1:43446
2016-07-21 08:11:46,045 [myid:] - INFO [WorkerReceiver[myid=5]:FastLeaderElection@600] - Notification: 1 (message format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,045 [myid:] - WARN [RecvWorker:5:QuorumCnxManager$RecvWorker@810] - Connection broken for id 5, my id = 1, error =
java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:392)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:795)
2016-07-21 08:11:46,047 [myid:] - WARN [RecvWorker:5:QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker
2016-07-21 08:11:46,045 [myid:] - WARN [RecvWorker:1:QuorumCnxManager$RecvWorker@810] - Connection broken for id 1, my id = 5, error =
java.net.SocketException: Socket closed
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at java.net.SocketInputStream.read(SocketInputStream.java:210)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:795)
2016-07-21 08:11:46,047 [myid:] - WARN [RecvWorker:1:QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker
2016-07-21 08:11:46,047 [myid:] - INFO [WorkerReceiver[myid=5]:FastLeaderElection@600] - Notification: 1 (message format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 4 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,047 [myid:] - WARN [SendWorker:5:QuorumCnxManager$SendWorker@727] - Interrupted while waiting for message on queue
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:879)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:65)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:715)
2016-07-21 08:11:46,048 [myid:] - WARN [SendWorker:5:QuorumCnxManager$SendWorker@736] - Send worker leaving thread
2016-07-21 08:11:46,046 [myid:] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 3 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,048 [myid:] - INFO [WorkerReceiver[myid=5]:FastLeaderElection@600] - Notification: 1 (message format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 3 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,048 [myid:] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 5 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,049 [myid:] - INFO [WorkerReceiver[myid=5]:FastLeaderElection@600] - Notification: 1 (message format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,049 [myid:] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 5 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,049 [myid:] - INFO [WorkerReceiver[myid=5]:FastLeaderElection@600] - Notification: 1 (message format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-07-21 08:11:46,245 [myid:] - INFO [QuorumPeer[myid=4]/0:0:0:0:0:0:0:0:11225:QuorumPeer@844] - FOLLOWING
2016-07-21 08:11:46,246 [myid:] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11223:QuorumPeer@844] - FOLLOWING
2016-07-21 08:11:46,247 [myid:] - INFO [QuorumPeer[myid=3]/0:0:0:0:0:0:0:0:11224:QuorumPeer@844] - FOLLOWING
2016-07-21 08:11:46,249 [myid:] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11222:QuorumPeer@844] - FOLLOWING
2016-07-21 08:11:46,249 [myid:] - INFO [QuorumPeer[myid=5]/0:0:0:0:0:0:0:0:11226:QuorumPeer@856] - LEADING
2016-07-21 08:11:46,251 [myid:] - INFO [QuorumPeer[myid=4]/0:0:0:0:0:0:0:0:11225:Learner@86] - TCP NoDelay set to: true
2016-07-21 08:11:46,255 [myid:] - INFO [QuorumPeer[myid=5]/0:0:0:0:0:0:0:0:11226:Leader@59] - TCP NoDelay set to: true
2016-07-21 08:11:46,259 [myid:] - INFO [QuorumPeer[myid=4]/0:0:0:0:0:0:0:0:11225:Environment@100] - Server environment:zookeeper.version=3.4.9-SNAPSHOT-1753645, built on 07/21/2016 07:46 GMT
2016-07-21 08:11:46,259 [myid:] - INFO [QuorumPeer[myid=4]/0:0:0:0:0:0:0:0:11225:Environment@100] - Server environment:host.name=asf907.gq1.ygridcore.net
2016-07-21 08:11:46,260 [myid:] - INFO [QuorumPeer[myid=4]/0:0:0:0:0:0:0:0:11225:Environment@100] - Server environment:java.version=1.7.0_80
2016-07-21 08:11:46,260 [myid:] - INFO [QuorumPeer[myid=4]/0:0:0:0:0:0:0:0:11225:Environment@100] - Server environment:java.vendor=Oracle Corporation
2016-07-21 08:11:46,260 [myid:] - INFO [QuorumPeer[myid=4]/0:0:0:0:0:0:0:0:11225:Environment@100] - Server environment:java.home=/home/jenkins/jenkins-slave/tools/hudson.model.JDK/latest1.7/jre
2016-07-21 08:11:46,260 [myid:] - INFO [QuorumPeer[myid=4]/0:0:0:0:0:0:0:0:11225:Environment@100] - Server environment:java.class.path=/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/test/classes:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/test/lib/antlr-2.7.6.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/test/lib/checkstyle-5.0.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/test/lib/commons-beanutils-core-1.7.0.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/test/lib/commons-cli-1.0.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/test/lib/commons-collections-3.2.2.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/test/lib/commons-lang-1.0.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/test/lib/commons-logging-1.0.3.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/test/lib/google-collections-0.9.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/test/lib/junit-4.8.1.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/test/lib/mockito-all-1.8.2.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/classes:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/src/java/lib/ivy-2.4.0.jar:/home/jenkins/tools/ant/latest/lib/ant.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/lib/jline-0.9.94.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/lib/log4j-1.2.16.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/lib/netty-3.10.5.Final.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/lib/slf4j-api-1.6.1.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/lib/slf4j-log4j12-1.6.1.jar:/home/jenkins/tools/ant/apache-ant-1.9.4/lib/ant-launcher.jar:/home/jenkins/tools/ant/latest/lib/ant-junit.jar:/home/jenkins/tools/ant/latest/lib/ant-junit4.jar
2016-07-21 08:11:46,260 [myid:] - INFO [QuorumPeer[myid=4]/0:0:0:0:0:0:0:0:11225:Environment@100] - Server environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2016-07-21 08:11:46,260 [myid:] - INFO [QuorumPeer[myid=4]/0:0:0:0:0:0:0:0:11225:Environment@100] - Server environment:java.io.tmpdir=/tmp
2016-07-21 08:11:46,260 [myid:] - INFO [QuorumPeer[myid=4]/0:0:0:0:0:0:0:0:11225:Environment@100] - Server environment:java.compiler=<NA>
2016-07-21 08:11:46,260 [myid:] - INFO [QuorumPeer[myid=4]/0:0:0:0:0:0:0:0:11225:Environment@100] - Server environment:os.name=Linux
2016-07-21 08:11:46,260 [myid:] - INFO [QuorumPeer[myid=4]/0:0:0:0:0:0:0:0:11225:Environment@100] - Server environment:os.arch=amd64
2016-07-21 08:11:46,260 [myid:] - INFO [QuorumPeer[myid=4]/0:0:0:0:0:0:0:0:11225:Environment@100] - Server environment:os.version=3.13.0-36-lowlatency
2016-07-21 08:11:46,260 [myid:] - INFO [QuorumPeer[myid=4]/0:0:0:
...[truncated 1420555 chars]...
ager$SendWorker.run(QuorumCnxManager.java:715)
2016-07-21 08:14:21,678 [myid:] - WARN [SendWorker:2:QuorumCnxManager$SendWorker@736] - Send worker leaving thread
2016-07-21 08:14:21,672 [myid:] - WARN [RecvWorker:1:QuorumCnxManager$RecvWorker@810] - Connection broken for id 1, my id = 5, error =
java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:392)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:795)
2016-07-21 08:14:21,679 [myid:] - WARN [RecvWorker:1:QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker
2016-07-21 08:14:21,672 [myid:] - WARN [SendWorker:5:QuorumCnxManager$SendWorker@727] - Interrupted while waiting for message on queue
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:879)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:65)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:715)
2016-07-21 08:14:21,679 [myid:] - WARN [SendWorker:5:QuorumCnxManager$SendWorker@736] - Send worker leaving thread
2016-07-21 08:14:21,679 [myid:] - WARN [SendWorker:1:QuorumCnxManager$SendWorker@727] - Interrupted while waiting for message on queue
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:879)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:65)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:715)
2016-07-21 08:14:21,679 [myid:] - WARN [SendWorker:1:QuorumCnxManager$SendWorker@736] - Send worker leaving thread
2016-07-21 08:14:21,678 [myid:] - WARN [SendWorker:1:QuorumCnxManager$SendWorker@727] - Interrupted while waiting for message on queue
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:879)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:65)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:715)
2016-07-21 08:14:21,679 [myid:] - WARN [SendWorker:1:QuorumCnxManager$SendWorker@736] - Send worker leaving thread
2016-07-21 08:14:21,677 [myid:] - WARN [SendWorker:1:QuorumCnxManager$SendWorker@727] - Interrupted while waiting for message on queue
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:879)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:65)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:715)
2016-07-21 08:14:21,680 [myid:] - WARN [SendWorker:1:QuorumCnxManager$SendWorker@736] - Send worker leaving thread
2016-07-21 08:14:21,675 [myid:] - WARN [SendWorker:1:QuorumCnxManager$SendWorker@727] - Interrupted while waiting for message on queue
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:879)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:65)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:715)
2016-07-21 08:14:21,680 [myid:] - WARN [SendWorker:1:QuorumCnxManager$SendWorker@736] - Send worker leaving thread
2016-07-21 08:14:22,000 [myid:] - INFO [SessionTracker:SessionTrackerImpl@162] - SessionTrackerImpl exited loop!
2016-07-21 08:14:22,317 [myid:] - INFO [/127.0.0.1:12241:QuorumCnxManager$Listener@560] - Leaving listener
2016-07-21 08:14:22,320 [myid:] - INFO [/127.0.0.1:12242:QuorumCnxManager$Listener@560] - Leaving listener
2016-07-21 08:14:22,579 [myid:] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11233:Follower@166] - shutdown called
java.lang.Exception: shutdown Follower
at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:166)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:850)
2016-07-21 08:14:22,579 [myid:] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11233:FollowerZooKeeperServer@140] - Shutting down
2016-07-21 08:14:22,579 [myid:] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11233:SyncRequestProcessor@209] - Shutting down
2016-07-21 08:14:22,579 [myid:] - WARN [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11233:QuorumPeer@874] - QuorumPeer main thread exited
2016-07-21 08:14:22,582 [myid:] - INFO [main:QuorumBase@306] - Shutting down quorum peer QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11234
2016-07-21 08:14:22,582 [myid:] - INFO [main:Follower@166] - shutdown called
java.lang.Exception: shutdown Follower
at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:166)
at org.apache.zookeeper.server.quorum.QuorumPeer.shutdown(QuorumPeer.java:891)
at org.apache.zookeeper.test.QuorumBase.shutdown(QuorumBase.java:307)
at org.apache.zookeeper.test.QuorumBase.shutdownServers(QuorumBase.java:298)
at org.apache.zookeeper.test.QuorumBase.tearDown(QuorumBase.java:285)
at org.apache.zookeeper.test.QuorumZxidSyncTest.tearDown(QuorumZxidSyncTest.java:169)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:37)
at org.junit.rules.TestWatchman$1.evaluate(TestWatchman.java:48)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:76)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184)
at org.junit.runners.ParentRunner.run(ParentRunner.java:236)
at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:39)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:532)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1179)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:1030)
2016-07-21 08:14:22,582 [myid:] - INFO [main:FollowerZooKeeperServer@140] - Shutting down
2016-07-21 08:14:22,582 [myid:] - INFO [main:ZooKeeperServer@469] - shutting down
2016-07-21 08:14:22,582 [myid:] - INFO [main:FollowerRequestProcessor@107] - Shutting down
2016-07-21 08:14:22,582 [myid:] - INFO [main:CommitProcessor@184] - Shutting down
2016-07-21 08:14:22,582 [myid:] - INFO [FollowerRequestProcessor:2:FollowerRequestProcessor@97] - FollowerRequestProcessor exited loop!
2016-07-21 08:14:22,582 [myid:] - INFO [CommitProcessor:2:CommitProcessor@153] - CommitProcessor exited loop!
2016-07-21 08:14:22,582 [myid:] - INFO [main:FinalRequestProcessor@402] - shutdown of request processor complete
2016-07-21 08:14:22,583 [myid:] - INFO [main:SyncRequestProcessor@209] - Shutting down
2016-07-21 08:14:22,583 [myid:] - INFO [SyncThread:2:SyncRequestProcessor@187] - SyncRequestProcessor exited!
2016-07-21 08:14:22,584 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11234:NIOServerCnxnFactory@219] - NIOServerCnxn factory exited run method
2016-07-21 08:14:22,585 [myid:] - WARN [SendWorker:3:QuorumCnxManager$SendWorker@727] - Interrupted while waiting for message on queue
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:879)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:65)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:715)
2016-07-21 08:14:22,585 [myid:] - WARN [RecvWorker:2:QuorumCnxManager$RecvWorker@810] - Connection broken for id 2, my id = 4, error =
java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:392)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:795)
2016-07-21 08:14:22,585 [myid:] - WARN [RecvWorker:2:QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker
2016-07-21 08:14:22,585 [myid:] - WARN [RecvWorker:3:QuorumCnxManager$RecvWorker@810] - Connection broken for id 3, my id = 2, error =
java.net.SocketException: Socket closed
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at java.net.SocketInputStream.read(SocketInputStream.java:210)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:795)
2016-07-21 08:14:22,586 [myid:] - WARN [RecvWorker:3:QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker
2016-07-21 08:14:22,585 [myid:] - WARN [RecvWorker:2:QuorumCnxManager$RecvWorker@810] - Connection broken for id 2, my id = 5, error =
java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:392)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:795)
2016-07-21 08:14:22,586 [myid:] - WARN [RecvWorker:2:QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker
2016-07-21 08:14:22,585 [myid:] - WARN [RecvWorker:5:QuorumCnxManager$RecvWorker@810] - Connection broken for id 5, my id = 2, error =
java.net.SocketException: Socket closed
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at java.net.SocketInputStream.read(SocketInputStream.java:210)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:795)
2016-07-21 08:14:22,587 [myid:] - WARN [RecvWorker:5:QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker
2016-07-21 08:14:22,585 [myid:] - WARN [SendWorker:3:QuorumCnxManager$SendWorker@736] - Send worker leaving thread
2016-07-21 08:14:22,585 [myid:] - WARN [RecvWorker:2:QuorumCnxManager$RecvWorker@810] - Connection broken for id 2, my id = 3, error =
java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:392)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:795)
2016-07-21 08:14:22,587 [myid:] - WARN [RecvWorker:2:QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker
2016-07-21 08:14:22,585 [myid:] - WARN [SendWorker:5:QuorumCnxManager$SendWorker@727] - Interrupted while waiting for message on queue
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:879)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:65)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:715)
2016-07-21 08:14:22,587 [myid:] - WARN [SendWorker:5:QuorumCnxManager$SendWorker@736] - Send worker leaving thread
2016-07-21 08:14:22,585 [myid:] - ERROR [/127.0.0.1:12239:QuorumCnxManager$Listener@547] - Exception while listening
java.net.SocketException: Socket closed
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
at java.net.ServerSocket.implAccept(ServerSocket.java:530)
at java.net.ServerSocket.accept(ServerSocket.java:498)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$Listener.run(QuorumCnxManager.java:539)
2016-07-21 08:14:22,587 [myid:] - WARN [SendWorker:2:QuorumCnxManager$SendWorker@727] - Interrupted while waiting for message on queue
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:879)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:65)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:715)
2016-07-21 08:14:22,588 [myid:] - WARN [SendWorker:2:QuorumCnxManager$SendWorker@736] - Send worker leaving thread
2016-07-21 08:14:22,587 [myid:] - WARN [SendWorker:2:QuorumCnxManager$SendWorker@727] - Interrupted while waiting for message on queue
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:879)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:65)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:715)
2016-07-21 08:14:22,589 [myid:] - WARN [SendWorker:2:QuorumCnxManager$SendWorker@736] - Send worker leaving thread
2016-07-21 08:14:22,586 [myid:] - WARN [SendWorker:4:QuorumCnxManager$SendWorker@727] - Interrupted while waiting for message on queue
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:879)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:65)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:715)
2016-07-21 08:14:22,589 [myid:] - WARN [SendWorker:4:QuorumCnxManager$SendWorker@736] - Send worker leaving thread
2016-07-21 08:14:22,586 [myid:] - WARN [RecvWorker:4:QuorumCnxManager$RecvWorker@810] - Connection broken for id 4, my id = 2, error =
java.net.SocketException: Socket closed
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at java.net.SocketInputStream.read(SocketInputStream.java:210)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:795)
2016-07-21 08:14:22,589 [myid:] - WARN [RecvWorker:4:QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker
2016-07-21 08:14:22,586 [myid:] - INFO [main:QuorumBase@310] - Shutting down leader election QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11234
2016-07-21 08:14:22,589 [myid:] - INFO [main:QuorumBase@315] - Waiting for QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11234 to exit thread
2016-07-21 08:14:22,585 [myid:] - WARN [SendWorker:2:QuorumCnxManager$SendWorker@727] - Interrupted while waiting for message on queue
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:879)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:65)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:715)
2016-07-21 08:14:22,590 [myid:] - WARN [SendWorker:2:QuorumCnxManager$SendWorker@736] - Send worker leaving thread
2016-07-21 08:14:22,673 [myid:] - INFO [/127.0.0.1:12238:QuorumCnxManager$Listener@560] - Leaving listener
2016-07-21 08:14:23,079 [myid:] - INFO [WorkerSender[myid=3]:FastLeaderElection$Messenger$WorkerSender@438] - WorkerSender is down
2016-07-21 08:14:23,080 [myid:] - INFO [WorkerSender[myid=4]:FastLeaderElection$Messenger$WorkerSender@438] - WorkerSender is down
2016-07-21 08:14:23,084 [myid:] - INFO [WorkerSender[myid=5]:FastLeaderElection$Messenger$WorkerSender@438] - WorkerSender is down
2016-07-21 08:14:23,088 [myid:] - INFO [WorkerReceiver[myid=3]:FastLeaderElection$Messenger$WorkerReceiver@407] - WorkerReceiver is down
2016-07-21 08:14:23,088 [myid:] - INFO [WorkerReceiver[myid=4]:FastLeaderElection$Messenger$WorkerReceiver@407] - WorkerReceiver is down
2016-07-21 08:14:23,089 [myid:] - INFO [WorkerReceiver[myid=5]:FastLeaderElection$Messenger$WorkerReceiver@407] - WorkerReceiver is down
2016-07-21 08:14:23,578 [myid:] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11234:Follower@166] - shutdown called
java.lang.Exception: shutdown Follower
at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:166)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:850)
2016-07-21 08:14:23,578 [myid:] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11234:FollowerZooKeeperServer@140] - Shutting down
2016-07-21 08:14:23,578 [myid:] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11234:SyncRequestProcessor@209] - Shutting down
2016-07-21 08:14:23,578 [myid:] - WARN [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11234:QuorumPeer@874] - QuorumPeer main thread exited
2016-07-21 08:14:23,579 [myid:] - INFO [main:QuorumBase@306] - Shutting down quorum peer QuorumPeer[myid=3]/0:0:0:0:0:0:0:0:11235
2016-07-21 08:14:23,579 [myid:] - INFO [main:Follower@166] - shutdown called
java.lang.Exception: shutdown Follower
at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:166)
at org.apache.zookeeper.server.quorum.QuorumPeer.shutdown(QuorumPeer.java:891)
at org.apache.zookeeper.test.QuorumBase.shutdown(QuorumBase.java:307)
at org.apache.zookeeper.test.QuorumBase.shutdownServers(QuorumBase.java:299)
at org.apache.zookeeper.test.QuorumBase.tearDown(QuorumBase.java:285)
at org.apache.zookeeper.test.QuorumZxidSyncTest.tearDown(QuorumZxidSyncTest.java:169)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:37)
at org.junit.rules.TestWatchman$1.evaluate(TestWatchman.java:48)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:76)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184)
at org.junit.runners.ParentRunner.run(ParentRunner.java:236)
at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:39)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:532)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1179)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:1030)
2016-07-21 08:14:23,579 [myid:] - INFO [main:FollowerZooKeeperServer@140] - Shutting down
2016-07-21 08:14:23,579 [myid:] - INFO [main:ZooKeeperServer@469] - shutting down
2016-07-21 08:14:23,580 [myid:] - INFO [main:FollowerRequestProcessor@107] - Shutting down
2016-07-21 08:14:23,580 [myid:] - INFO [main:CommitProcessor@184] - Shutting down
2016-07-21 08:14:23,580 [myid:] - INFO [FollowerRequestProcessor:3:FollowerRequestProcessor@97] - FollowerRequestProcessor exited loop!
2016-07-21 08:14:23,580 [myid:] - INFO [CommitProcessor:3:CommitProcessor@153] - CommitProcessor exited loop!
2016-07-21 08:14:23,580 [myid:] - INFO [main:FinalRequestProcessor@402] - shutdown of request processor complete
2016-07-21 08:14:23,581 [myid:] - INFO [main:SyncRequestProcessor@209] - Shutting down
2016-07-21 08:14:23,581 [myid:] - INFO [SyncThread:3:SyncRequestProcessor@187] - SyncRequestProcessor exited!
2016-07-21 08:14:23,582 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11235:NIOServerCnxnFactory@219] - NIOServerCnxn factory exited run method
2016-07-21 08:14:23,583 [myid:] - ERROR [/127.0.0.1:12240:QuorumCnxManager$Listener@547] - Exception while listening
java.net.SocketException: Socket closed
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
at java.net.ServerSocket.implAccept(ServerSocket.java:530)
at java.net.ServerSocket.accept(ServerSocket.java:498)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$Listener.run(QuorumCnxManager.java:539)
2016-07-21 08:14:23,583 [myid:] - WARN [SendWorker:5:QuorumCnxManager$SendWorker@727] - Interrupted while waiting for message on queue
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:879)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:65)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:715)
2016-07-21 08:14:23,585 [myid:] - WARN [SendWorker:5:QuorumCnxManager$SendWorker@736] - Send worker leaving thread
2016-07-21 08:14:23,584 [myid:] - WARN [RecvWorker:4:QuorumCnxManager$RecvWorker@810] - Connection broken for id 4, my id = 3, error =
java.net.SocketException: Socket closed
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at java.net.SocketInputStream.read(SocketInputStream.java:210)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:795)
2016-07-21 08:14:23,585 [myid:] - WARN [RecvWorker:4:QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker
2016-07-21 08:14:23,584 [myid:] - WARN [SendWorker:4:QuorumCnxManager$SendWorker@727] - Interrupted while waiting for message on queue
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:879)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:65)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:715)
2016-07-21 08:14:23,585 [myid:] - WARN [SendWorker:4:QuorumCnxManager$SendWorker@736] - Send worker leaving thread
2016-07-21 08:14:23,584 [myid:] - WARN [RecvWorker:5:QuorumCnxManager$RecvWorker@810] - Connection broken for id 5, my id = 3, error =
java.net.SocketException: Socket closed
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at java.net.SocketInputStream.read(SocketInputStream.java:210)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:795)
2016-07-21 08:14:23,585 [myid:] - WARN [RecvWorker:5:QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker
2016-07-21 08:14:23,584 [myid:] - WARN [RecvWorker:3:QuorumCnxManager$RecvWorker@810] - Connection broken for id 3, my id = 5, error =
java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:392)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:795)
2016-07-21 08:14:23,586 [myid:] - WARN [RecvWorker:3:QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker
2016-07-21 08:14:23,584 [myid:] - WARN [RecvWorker:3:QuorumCnxManager$RecvWorker@810] - Connection broken for id 3, my id = 4, error =
java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:392)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:795)
2016-07-21 08:14:23,586 [myid:] - WARN [RecvWorker:3:QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker
2016-07-21 08:14:23,584 [myid:] - INFO [main:QuorumBase@310] - Shutting down leader election QuorumPeer[myid=3]/0:0:0:0:0:0:0:0:11235
2016-07-21 08:14:23,586 [myid:] - INFO [main:QuorumBase@315] - Waiting for QuorumPeer[myid=3]/0:0:0:0:0:0:0:0:11235 to exit thread
2016-07-21 08:14:23,586 [myid:] - WARN [SendWorker:3:QuorumCnxManager$SendWorker@727] - Interrupted while waiting for message on queue
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:879)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:65)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:715)
2016-07-21 08:14:23,586 [myid:] - WARN [SendWorker:3:QuorumCnxManager$SendWorker@736] - Send worker leaving thread
2016-07-21 08:14:23,586 [myid:] - WARN [SendWorker:3:QuorumCnxManager$SendWorker@727] - Interrupted while waiting for message on queue
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:879)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:65)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:715)
2016-07-21 08:14:23,587 [myid:] - WARN [SendWorker:3:QuorumCnxManager$SendWorker@736] - Send worker leaving thread
2016-07-21 08:14:23,588 [myid:] - INFO [/127.0.0.1:12239:QuorumCnxManager$Listener@560] - Leaving listener
2016-07-21 08:14:24,355 [myid:] - INFO [WorkerSender[myid=2]:FastLeaderElection$Messenger$WorkerSender@438] - WorkerSender is down
2016-07-21 08:14:24,358 [myid:] - INFO [WorkerSender[myid=1]:FastLeaderElection$Messenger$WorkerSender@438] - WorkerSender is down
2016-07-21 08:14:24,360 [myid:] - INFO [WorkerSender[myid=3]:FastLeaderElection$Messenger$WorkerSender@438] - WorkerSender is down
2016-07-21 08:14:24,361 [myid:] - INFO [WorkerReceiver[myid=1]:FastLeaderElection$Messenger$WorkerReceiver@407] - WorkerReceiver is down
2016-07-21 08:14:24,361 [myid:] - INFO [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@407] - WorkerReceiver is down
2016-07-21 08:14:24,362 [myid:] - INFO [WorkerReceiver[myid=3]:FastLeaderElection$Messenger$WorkerReceiver@407] - WorkerReceiver is down
2016-07-21 08:14:24,578 [myid:] - INFO [QuorumPeer[myid=3]/0:0:0:0:0:0:0:0:11235:Follower@166] - shutdown called
java.lang.Exception: shutdown Follower
at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:166)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:850)
2016-07-21 08:14:24,578 [myid:] - INFO [QuorumPeer[myid=3]/0:0:0:0:0:0:0:0:11235:FollowerZooKeeperServer@140] - Shutting down
2016-07-21 08:14:24,578 [myid:] - INFO [QuorumPeer[myid=3]/0:0:0:0:0:0:0:0:11235:SyncRequestProcessor@209] - Shutting down
2016-07-21 08:14:24,578 [myid:] - WARN [QuorumPeer[myid=3]/0:0:0:0:0:0:0:0:11235:QuorumPeer@874] - QuorumPeer main thread exited
2016-07-21 08:14:24,579 [myid:] - INFO [main:QuorumBase@306] - Shutting down quorum peer QuorumPeer[myid=4]/0:0:0:0:0:0:0:0:11236
2016-07-21 08:14:24,579 [myid:] - INFO [main:Follower@166] - shutdown called
java.lang.Exception: shutdown Follower
at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:166)
at org.apache.zookeeper.server.quorum.QuorumPeer.shutdown(QuorumPeer.java:891)
at org.apache.zookeeper.test.QuorumBase.shutdown(QuorumBase.java:307)
at org.apache.zookeeper.test.QuorumBase.shutdownServers(QuorumBase.java:300)
at org.apache.zookeeper.test.QuorumBase.tearDown(QuorumBase.java:285)
at org.apache.zookeeper.test.QuorumZxidSyncTest.tearDown(QuorumZxidSyncTest.java:169)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:37)
at org.junit.rules.TestWatchman$1.evaluate(TestWatchman.java:48)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:76)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184)
at org.junit.runners.ParentRunner.run(ParentRunner.java:236)
at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:39)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:532)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1179)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:1030)
2016-07-21 08:14:24,580 [myid:] - INFO [main:FollowerZooKeeperServer@140] - Shutting down
2016-07-21 08:14:24,581 [myid:] - INFO [main:ZooKeeperServer@469] - shutting down
2016-07-21 08:14:24,581 [myid:] - INFO [main:FollowerRequestProcessor@107] - Shutting down
2016-07-21 08:14:24,581 [myid:] - INFO [main:CommitProcessor@184] - Shutting down
2016-07-21 08:14:24,581 [myid:] - INFO [main:FinalRequestProcessor@402] - shutdown of request processor complete
2016-07-21 08:14:24,581 [myid:] - INFO [FollowerRequestProcessor:4:FollowerRequestProcessor@97] - FollowerRequestProcessor exited loop!
2016-07-21 08:14:24,581 [myid:] - INFO [main:SyncRequestProcessor@209] - Shutting down
2016-07-21 08:14:24,581 [myid:] - INFO [CommitProcessor:4:CommitProcessor@153] - CommitProcessor exited loop!
2016-07-21 08:14:24,582 [myid:] - INFO [SyncThread:4:SyncRequestProcessor@187] - SyncRequestProcessor exited!
2016-07-21 08:14:24,583 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11236:NIOServerCnxnFactory@219] - NIOServerCnxn factory exited run method
2016-07-21 08:14:24,585 [myid:] - INFO [main:QuorumBase@310] - Shutting down leader election QuorumPeer[myid=4]/0:0:0:0:0:0:0:0:11236
2016-07-21 08:14:24,586 [myid:] - ERROR [/127.0.0.1:12241:QuorumCnxManager$Listener@547] - Exception while listening
java.net.SocketException: Socket closed
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
at java.net.ServerSocket.implAccept(ServerSocket.java:530)
at java.net.ServerSocket.accept(ServerSocket.java:498)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$Listener.run(QuorumCnxManager.java:539)
2016-07-21 08:14:24,586 [myid:] - WARN [SendWorker:5:QuorumCnxManager$SendWorker@727] - Interrupted while waiting for message on queue
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:879)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:65)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:715)
2016-07-21 08:14:24,586 [myid:] - WARN [SendWorker:5:QuorumCnxManager$SendWorker@736] - Send worker leaving thread
2016-07-21 08:14:24,586 [myid:] - INFO [main:QuorumBase@315] - Waiting for QuorumPeer[myid=4]/0:0:0:0:0:0:0:0:11236 to exit thread
2016-07-21 08:14:24,586 [myid:] - WARN [RecvWorker:4:QuorumCnxManager$RecvWorker@810] - Connection broken for id 4, my id = 5, error =
java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:392)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:795)
2016-07-21 08:14:24,587 [myid:] - WARN [RecvWorker:4:QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker
2016-07-21 08:14:24,585 [myid:] - INFO [/127.0.0.1:12240:QuorumCnxManager$Listener@560] - Leaving listener
2016-07-21 08:14:24,585 [myid:] - WARN [RecvWorker:5:QuorumCnxManager$RecvWorker@810] - Connection broken for id 5, my id = 4, error =
java.net.SocketException: Socket closed
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at java.net.SocketInputStream.read(SocketInputStream.java:210)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:795)
2016-07-21 08:14:24,588 [myid:] - WARN [RecvWorker:5:QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker
2016-07-21 08:14:24,588 [myid:] - WARN [SendWorker:4:QuorumCnxManager$SendWorker@727] - Interrupted while waiting for message on queue
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:879)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:65)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:715)
2016-07-21 08:14:24,588 [myid:] - WARN [SendWorker:4:QuorumCnxManager$SendWorker@736] - Send worker leaving thread
2016-07-21 08:14:25,579 [myid:] - INFO [QuorumPeer[myid=4]/0:0:0:0:0:0:0:0:11236:Follower@166] - shutdown called
java.lang.Exception: shutdown Follower
at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:166)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:850)
2016-07-21 08:14:25,579 [myid:] - INFO [QuorumPeer[myid=4]/0:0:0:0:0:0:0:0:11236:FollowerZooKeeperServer@140] - Shutting down
2016-07-21 08:14:25,579 [myid:] - INFO [QuorumPeer[myid=4]/0:0:0:0:0:0:0:0:11236:SyncRequestProcessor@209] - Shutting down
2016-07-21 08:14:25,579 [myid:] - WARN [QuorumPeer[myid=4]/0:0:0:0:0:0:0:0:11236:QuorumPeer@874] - QuorumPeer main thread exited
2016-07-21 08:14:25,579 [myid:] - INFO [main:QuorumBase@306] - Shutting down quorum peer QuorumPeer[myid=5]/0:0:0:0:0:0:0:0:11237
2016-07-21 08:14:25,579 [myid:] - INFO [main:Leader@496] - Shutting down
2016-07-21 08:14:25,580 [myid:] - INFO [main:Leader@502] - Shutdown called
java.lang.Exception: shutdown Leader! reason: quorum Peer shutdown
at org.apache.zookeeper.server.quorum.Leader.shutdown(Leader.java:502)
at org.apache.zookeeper.server.quorum.QuorumPeer.shutdown(QuorumPeer.java:888)
at org.apache.zookeeper.test.QuorumBase.shutdown(QuorumBase.java:307)
at org.apache.zookeeper.test.QuorumBase.shutdownServers(QuorumBase.java:301)
at org.apache.zookeeper.test.QuorumBase.tearDown(QuorumBase.java:285)
at org.apache.zookeeper.test.QuorumZxidSyncTest.tearDown(QuorumZxidSyncTest.java:169)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:37)
at org.junit.rules.TestWatchman$1.evaluate(TestWatchman.java:48)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:76)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184)
at org.junit.runners.ParentRunner.run(ParentRunner.java:236)
at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:39)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:532)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1179)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:1030)
2016-07-21 08:14:25,581 [myid:] - INFO [main:ZooKeeperServer@469] - shutting down
2016-07-21 08:14:25,581 [myid:] - INFO [LearnerCnxAcceptor-/127.0.0.1:12237:Leader$LearnerCnxAcceptor@325] - exception while shutting down acceptor: java.net.SocketException: Socket closed
2016-07-21 08:14:25,582 [myid:] - INFO [main:SessionTrackerImpl@225] - Shutting down
2016-07-21 08:14:25,582 [myid:] - INFO [main:PrepRequestProcessor@765] - Shutting down
2016-07-21 08:14:25,582 [myid:] - INFO [main:ProposalRequestProcessor@88] - Shutting down
2016-07-21 08:14:25,582 [myid:] - INFO [main:CommitProcessor@184] - Shutting down
2016-07-21 08:14:25,582 [myid:] - INFO [ProcessThread(sid:5 cport:-1)::PrepRequestProcessor@143] - PrepRequestProcessor exited loop!
2016-07-21 08:14:25,583 [myid:] - INFO [CommitProcessor:5:CommitProcessor@153] - CommitProcessor exited loop!
2016-07-21 08:14:25,582 [myid:] - INFO [main:Leader$ToBeAppliedRequestProcessor@661] - Shutting down
2016-07-21 08:14:25,583 [myid:] - INFO [main:FinalRequestProcessor@402] - shutdown of request processor complete
2016-07-21 08:14:25,583 [myid:] - INFO [main:SyncRequestProcessor@209] - Shutting down
2016-07-21 08:14:25,583 [myid:] - INFO [SyncThread:5:SyncRequestProcessor@187] - SyncRequestProcessor exited!
2016-07-21 08:14:25,585 [myid:] - WARN [LearnerHandler-/127.0.0.1:43485:LearnerHandler@644] - ******* GOODBYE /127.0.0.1:43485 ********
2016-07-21 08:14:25,585 [myid:] - WARN [LearnerHandler-/127.0.0.1:43486:LearnerHandler@644] - ******* GOODBYE /127.0.0.1:43486 ********
2016-07-21 08:14:25,585 [myid:] - WARN [LearnerHandler-/127.0.0.1:43486:LearnerHandler@656] - Ignoring unexpected exception
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1219)
at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:340)
at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:338)
at org.apache.zookeeper.server.quorum.LearnerHandler.shutdown(LearnerHandler.java:654)
at org.apache.zookeeper.server.quorum.LearnerHandler.run(LearnerHandler.java:647)
2016-07-21 08:14:25,585 [myid:] - WARN [LearnerHandler-/127.0.0.1:43485:LearnerHandler@656] - Ignoring unexpected exception
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1219)
at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:340)
at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:338)
at org.apache.zookeeper.server.quorum.LearnerHandler.shutdown(LearnerHandler.java:654)
at org.apache.zookeeper.server.quorum.LearnerHandler.run(LearnerHandler.java:647)
2016-07-21 08:14:25,585 [myid:] - WARN [LearnerHandler-/127.0.0.1:43488:LearnerHandler@644] - ******* GOODBYE /127.0.0.1:43488 ********
2016-07-21 08:14:25,586 [myid:] - INFO [/127.0.0.1:12241:QuorumCnxManager$Listener@560] - Leaving listener
2016-07-21 08:14:25,586 [myid:] - WARN [LearnerHandler-/127.0.0.1:43487:LearnerHandler@644] - ******* GOODBYE /127.0.0.1:43487 ********
2016-07-21 08:14:25,586 [myid:] - WARN [LearnerHandler-/127.0.0.1:43487:LearnerHandler@656] - Ignoring unexpected exception
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1219)
at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:340)
at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:338)
at org.apache.zookeeper.server.quorum.LearnerHandler.shutdown(LearnerHandler.java:654)
at org.apache.zookeeper.server.quorum.LearnerHandler.run(LearnerHandler.java:647)
2016-07-21 08:14:25,586 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11237:NIOServerCnxnFactory@219] - NIOServerCnxn factory exited run method
2016-07-21 08:14:25,586 [myid:] - WARN [LearnerHandler-/127.0.0.1:43488:LearnerHandler@656] - Ignoring unexpected exception
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1219)
at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:340)
at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:338)
at org.apache.zookeeper.server.quorum.LearnerHandler.shutdown(LearnerHandler.java:654)
at org.apache.zookeeper.server.quorum.LearnerHandler.run(LearnerHandler.java:647)
2016-07-21 08:14:25,587 [myid:] - WARN [QuorumPeer[myid=5]/0:0:0:0:0:0:0:0:11237:QuorumPeer@862] - Unexpected exception
java.lang.InterruptedException: sleep interrupted
at java.lang.Thread.sleep(Native Method)
at org.apache.zookeeper.server.quorum.Leader.lead(Leader.java:456)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:859)
2016-07-21 08:14:25,588 [myid:] - INFO [QuorumPeer[myid=5]/0:0:0:0:0:0:0:0:11237:Leader@496] - Shutting down
2016-07-21 08:14:25,588 [myid:] - INFO [main:QuorumBase@310] - Shutting down leader election QuorumPeer[myid=5]/0:0:0:0:0:0:0:0:11237
2016-07-21 08:14:25,588 [myid:] - ERROR [/127.0.0.1:12242:QuorumCnxManager$Listener@547] - Exception while listening
java.net.SocketException: Socket closed
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
at java.net.ServerSocket.implAccept(ServerSocket.java:530)
at java.net.ServerSocket.accept(ServerSocket.java:498)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$Listener.run(QuorumCnxManager.java:539)
2016-07-21 08:14:25,588 [myid:] - INFO [main:QuorumBase@315] - Waiting for QuorumPeer[myid=5]/0:0:0:0:0:0:0:0:11237 to exit thread
2016-07-21 08:14:25,588 [myid:] - WARN [QuorumPeer[myid=5]/0:0:0:0:0:0:0:0:11237:QuorumPeer@874] - QuorumPeer main thread exited
2016-07-21 08:14:25,589 [myid:] - INFO [main:FourLetterWordMain@62] - connecting to 127.0.0.1 11233
2016-07-21 08:14:25,589 [myid:] - INFO [main:QuorumBase@291] - 127.0.0.1:11233 is no longer accepting client connections
2016-07-21 08:14:25,589 [myid:] - INFO [main:FourLetterWordMain@62] - connecting to 127.0.0.1 11234
2016-07-21 08:14:25,589 [myid:] - INFO [main:QuorumBase@291] - 127.0.0.1:11234 is no longer accepting client connections
2016-07-21 08:14:25,590 [myid:] - INFO [main:FourLetterWordMain@62] - connecting to 127.0.0.1 11235
2016-07-21 08:14:25,590 [myid:] - INFO [main:QuorumBase@291] - 127.0.0.1:11235 is no longer accepting client connections
2016-07-21 08:14:25,590 [myid:] - INFO [main:FourLetterWordMain@62] - connecting to 127.0.0.1 11236
2016-07-21 08:14:25,590 [myid:] - INFO [main:QuorumBase@291] - 127.0.0.1:11236 is no longer accepting client connections
2016-07-21 08:14:25,590 [myid:] - INFO [main:FourLetterWordMain@62] - connecting to 127.0.0.1 11237
2016-07-21 08:14:25,590 [myid:] - INFO [main:QuorumBase@291] - 127.0.0.1:11237 is no longer accepting client connections
2016-07-21 08:14:25,592 [myid:] - INFO [main:ZKTestCase$1@60] - SUCCEEDED testLateLogs
2016-07-21 08:14:25,592 [myid:] - INFO [main:ZKTestCase$1@55] - FINISHED testLateLogs
{noformat}
flaky, flaky-test 9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
1 year, 21 weeks ago 0|i31cmf:
ZooKeeper ZOOKEEPER-2485

ZOOKEEPER-3170 Flaky Test: org.apache.zookeeper.test.FourLetterWordsTest.testFourLetterWords

Sub-task Closed Major Cannot Reproduce Andor Molnar Michael Han Michael Han 21/Jul/16 18:40   19/Dec/19 18:01 25/Oct/18 10:38 3.4.8 3.5.5 tests   0 2   From https://builds.apache.org/job/ZooKeeper_branch34_jdk7/1156/

{noformat}
Error Message

test timed out after 30000 milliseconds
Stacktrace

java.lang.Exception: test timed out after 30000 milliseconds
at java.io.FileOutputStream.writeBytes(Native Method)
at java.io.FileOutputStream.write(FileOutputStream.java:345)
at java.io.BufferedOutputStream.write(BufferedOutputStream.java:122)
at java.io.PrintStream.write(PrintStream.java:480)
at java.io.PrintStream.write(PrintStream.java:480)
at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)
at sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:291)
at sun.nio.cs.StreamEncoder.implFlush(StreamEncoder.java:295)
at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:141)
at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:229)
at org.apache.log4j.helpers.QuietWriter.flush(QuietWriter.java:59)
at org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:324)
at org.apache.log4j.WriterAppender.append(WriterAppender.java:162)
at org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251)
at org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:66)
at org.apache.log4j.Category.callAppenders(Category.java:206)
at org.apache.log4j.Category.forcedLog(Category.java:391)
at org.apache.log4j.Category.log(Category.java:856)
at org.slf4j.impl.Log4jLoggerAdapter.info(Log4jLoggerAdapter.java:305)
at org.apache.zookeeper.test.FourLetterWordsTest.verify(FourLetterWordsTest.java:121)
at org.apache.zookeeper.test.FourLetterWordsTest.testFourLetterWords(FourLetterWordsTest.java:52)
at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:55)
Standard Output

2016-07-21 08:05:43,150 [myid:] - INFO [main:PortAssignment@32] - assigning port 11221
2016-07-21 08:05:43,156 [myid:] - INFO [main:ZKTestCase$1@50] - STARTING testDisconnectedAddAuth
2016-07-21 08:05:43,157 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@53] - RUNNING TEST METHOD testDisconnectedAddAuth
2016-07-21 08:05:43,178 [myid:] - INFO [main:Environment@100] - Server environment:zookeeper.version=3.4.9-SNAPSHOT-1753645, built on 07/21/2016 07:46 GMT
2016-07-21 08:05:43,178 [myid:] - INFO [main:Environment@100] - Server environment:host.name=asf907.gq1.ygridcore.net
2016-07-21 08:05:43,178 [myid:] - INFO [main:Environment@100] - Server environment:java.version=1.7.0_80
2016-07-21 08:05:43,178 [myid:] - INFO [main:Environment@100] - Server environment:java.vendor=Oracle Corporation
2016-07-21 08:05:43,178 [myid:] - INFO [main:Environment@100] - Server environment:java.home=/home/jenkins/jenkins-slave/tools/hudson.model.JDK/latest1.7/jre
2016-07-21 08:05:43,179 [myid:] - INFO [main:Environment@100] - Server environment:java.class.path=/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/test/classes:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/test/lib/antlr-2.7.6.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/test/lib/checkstyle-5.0.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/test/lib/commons-beanutils-core-1.7.0.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/test/lib/commons-cli-1.0.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/test/lib/commons-collections-3.2.2.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/test/lib/commons-lang-1.0.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/test/lib/commons-logging-1.0.3.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/test/lib/google-collections-0.9.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/test/lib/junit-4.8.1.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/test/lib/mockito-all-1.8.2.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/classes:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/src/java/lib/ivy-2.4.0.jar:/home/jenkins/tools/ant/latest/lib/ant.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/lib/jline-0.9.94.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/lib/log4j-1.2.16.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/lib/netty-3.10.5.Final.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/lib/slf4j-api-1.6.1.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/lib/slf4j-log4j12-1.6.1.jar:/home/jenkins/tools/ant/apache-ant-1.9.4/lib/ant-launcher.jar:/home/jenkins/tools/ant/latest/lib/ant-junit.jar:/home/jenkins/tools/ant/latest/lib/ant-junit4.jar
2016-07-21 08:05:43,179 [myid:] - INFO [main:Environment@100] - Server environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2016-07-21 08:05:43,179 [myid:] - INFO [main:Environment@100] - Server environment:java.io.tmpdir=/tmp
2016-07-21 08:05:43,179 [myid:] - INFO [main:Environment@100] - Server environment:java.compiler=<NA>
2016-07-21 08:05:43,179 [myid:] - INFO [main:Environment@100] - Server environment:os.name=Linux
2016-07-21 08:05:43,180 [myid:] - INFO [main:Environment@100] - Server environment:os.arch=amd64
2016-07-21 08:05:43,180 [myid:] - INFO [main:Environment@100] - Server environment:os.version=3.13.0-36-lowlatency
2016-07-21 08:05:43,180 [myid:] - INFO [main:Environment@100] - Server environment:user.name=jenkins
2016-07-21 08:05:43,182 [myid:] - INFO [main:Environment@100] - Server environment:user.home=/home/jenkins
2016-07-21 08:05:43,182 [myid:] - INFO [main:Environment@100] - Server environment:user.dir=/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4
2016-07-21 08:05:43,199 [myid:] - INFO [main:ZooKeeperServer@170] - Created server with tickTime 3000 minSessionTimeout 6000 maxSessionTimeout 60000 datadir /home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/test/tmp/test4168931007806197633.junit.dir/version-2 snapdir /home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/test/tmp/test4168931007806197633.junit.dir/version-2
2016-07-21 08:05:43,295 [myid:] - INFO [main:NettyServerCnxnFactory@365] - binding to port 0.0.0.0/0.0.0.0:11221
2016-07-21 08:05:43,387 [myid:] - INFO [main:ACLTest@62] - starting up the zookeeper server .. waiting
2016-07-21 08:05:43,388 [myid:] - INFO [main:FourLetterWordMain@62] - connecting to 127.0.0.1 11221
2016-07-21 08:05:43,420 [myid:] - INFO [New I/O worker #1:NettyServerCnxn@632] - Processing stat command from /127.0.0.1:35232
2016-07-21 08:05:43,422 [myid:] - INFO [New I/O worker #1:NettyServerCnxn$StatCommand@469] - Stat command output
2016-07-21 08:05:43,431 [myid:] - INFO [main:Environment@100] - Client environment:zookeeper.version=3.4.9-SNAPSHOT-1753645, built on 07/21/2016 07:46 GMT
2016-07-21 08:05:43,431 [myid:] - INFO [main:Environment@100] - Client environment:host.name=asf907.gq1.ygridcore.net
2016-07-21 08:05:43,431 [myid:] - INFO [main:Environment@100] - Client environment:java.version=1.7.0_80
2016-07-21 08:05:43,431 [myid:] - INFO [main:Environment@100] - Client environment:java.vendor=Oracle Corporation
2016-07-21 08:05:43,431 [myid:] - INFO [main:Environment@100] - Client environment:java.home=/home/jenkins/jenkins-slave/tools/hudson.model.JDK/latest1.7/jre
2016-07-21 08:05:43,431 [myid:] - INFO [main:Environment@100] - Client environment:java.class.path=/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/test/classes:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/test/lib/antlr-2.7.6.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/test/lib/checkstyle-5.0.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/test/lib/commons-beanutils-core-1.7.0.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/test/lib/commons-cli-1.0.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/test/lib/commons-collections-3.2.2.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/test/lib/commons-lang-1.0.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/test/lib/commons-logging-1.0.3.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/test/lib/google-collections-0.9.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/test/lib/junit-4.8.1.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/test/lib/mockito-all-1.8.2.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/classes:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/src/java/lib/ivy-2.4.0.jar:/home/jenkins/tools/ant/latest/lib/ant.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/lib/jline-0.9.94.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/lib/log4j-1.2.16.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/lib/netty-3.10.5.Final.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/lib/slf4j-api-1.6.1.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/lib/slf4j-log4j12-1.6.1.jar:/home/jenkins/tools/ant/apache-ant-1.9.4/lib/ant-launcher.jar:/home/jenkins/tools/ant/latest/lib/ant-junit.jar:/home/jenkins/tools/ant/latest/lib/ant-junit4.jar
2016-07-21 08:05:43,432 [myid:] - INFO [main:Environment@100] - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2016-07-21 08:05:43,432 [myid:] - INFO [main:Environment@100] - Client environment:java.io.tmpdir=/tmp
2016-07-21 08:05:43,432 [myid:] - INFO [main:Environment@100] - Client environment:java.compiler=<NA>
2016-07-21 08:05:43,432 [myid:] - INFO [main:Environment@100] - Client environment:os.name=Linux
2016-07-21 08:05:43,432 [myid:] - INFO [main:Environment@100] - Client environment:os.arch=amd64
2016-07-21 08:05:43,433 [myid:] - INFO [main:Environment@100] - Client environment:os.version=3.13.0-36-lowlatency
2016-07-21 08:05:43,433 [myid:] - INFO [main:Environment@100] - Client environment:user.name=jenkins
2016-07-21 08:05:43,433 [myid:] - INFO [main:Environment@100] - Client environment:user.home=/home/jenkins
2016-07-21 08:05:43,433 [myid:] - INFO [main:Environment@100] - Client environment:user.dir=/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4
2016-07-21 08:05:43,434 [myid:] - INFO [main:ZooKeeper@438] - Initiating client connection, connectString=127.0.0.1:11221 sessionTimeout=30000 watcher=org.apache.zookeeper.test.ACLTest@14f8e8b
2016-07-21 08:05:43,450 [myid:] - INFO [main-SendThread(127.0.0.1:11221):ClientCnxn$SendThread@1032] - Opening socket connection to server 127.0.0.1/127.0.0.1:11221. Will not attempt to authenticate using SASL (unknown error)
2016-07-21 08:05:43,450 [myid:] - INFO [main-SendThread(127.0.0.1:11221):ClientCnxn$SendThread@876] - Socket connection established to 127.0.0.1/127.0.0.1:11221, initiating session
2016-07-21 08:05:43,454 [myid:] - INFO [New I/O worker #2:ZooKeeperServer@900] - Client attempting to establish new session at /127.0.0.1:35233
2016-07-21 08:05:43,458 [myid:] - INFO [SyncThread:0:FileTxnLog@203] - Creating new log file: log.1
2016-07-21 08:05:43,468 [myid:] - INFO [SyncThread:0:ZooKeeperServer@645] - Established session 0x1560c7d251c0000 with negotiated timeout 30000 for client /127.0.0.1:35233
2016-07-21 08:05:43,471 [myid:] - INFO [main-SendThread(127.0.0.1:11221):ClientCnxn$SendThread@1299] - Session establishment complete on server 127.0.0.1/127.0.0.1:11221, sessionid = 0x1560c7d251c0000, negotiated timeout = 30000
2016-07-21 08:05:43,474 [myid:] - INFO [main-EventThread:ACLTest@175] - Event:SyncConnected None null
2016-07-21 08:05:43,475 [myid:] - INFO [New I/O worker #2:ZooKeeperServer@924] - got auth packet /127.0.0.1:35233
2016-07-21 08:05:43,475 [myid:] - WARN [main-EventThread:ACLTest@182] - startsignal null
2016-07-21 08:05:43,478 [myid:] - INFO [New I/O worker #2:ZooKeeperServer@958] - auth success /127.0.0.1:35233
2016-07-21 08:05:43,479 [myid:] - INFO [New I/O worker #2:ZooKeeperServer@924] - got auth packet /127.0.0.1:35233
2016-07-21 08:05:43,479 [myid:] - INFO [New I/O worker #2:ZooKeeperServer@958] - auth success /127.0.0.1:35233
2016-07-21 08:05:43,485 [myid:] - INFO [ProcessThread(sid:0 cport:11221)::PrepRequestProcessor@487] - Processed session termination for sessionid: 0x1560c7d251c0000
2016-07-21 08:05:43,488 [myid:] - WARN [New I/O worker #2:NettyServerCnxnFactory$CnxnChannelHandler@111] - Exception caught [id: 0xbe54f7b5, /127.0.0.1:35233 :> /127.0.0.1:11221] EXCEPTION: java.nio.channels.ClosedChannelException
java.nio.channels.ClosedChannelException
at sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:270)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:479)
at org.jboss.netty.channel.socket.nio.SocketSendBufferPool$UnpooledSendBuffer.transferTo(SocketSendBufferPool.java:203)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:201)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromTaskLoop(AbstractNioWorker.java:151)
at org.jboss.netty.channel.socket.nio.AbstractNioChannel$WriteTask.run(AbstractNioChannel.java:315)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:391)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:315)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2016-07-21 08:05:43,587 [myid:] - INFO [main:ZooKeeper@684] - Session: 0x1560c7d251c0000 closed
2016-07-21 08:05:43,588 [myid:] - INFO [main-EventThread:ClientCnxn$EventThread@519] - EventThread shut down for session: 0x1560c7d251c0000
2016-07-21 08:05:43,588 [myid:] - INFO [main:NettyServerCnxnFactory@342] - shutdown called 0.0.0.0/0.0.0.0:11221
2016-07-21 08:05:43,607 [myid:] - INFO [main:ZooKeeperServer@469] - shutting down
2016-07-21 08:05:43,607 [myid:] - INFO [main:SessionTrackerImpl@225] - Shutting down
2016-07-21 08:05:43,607 [myid:] - INFO [main:PrepRequestProcessor@765] - Shutting down
2016-07-21 08:05:43,608 [myid:] - INFO [main:SyncRequestProcessor@209] - Shutting down
2016-07-21 08:05:43,608 [myid:] - INFO [ProcessThread(sid:0 cport:11221)::PrepRequestProcessor@143] - PrepRequestProcessor exited loop!
2016-07-21 08:05:43,608 [myid:] - INFO [SyncThread:0:SyncRequestProcessor@187] - SyncRequestProcessor exited!
2016-07-21 08:05:43,608 [myid:] - INFO [main:FinalRequestProcessor@402] - shutdown of request processor complete
2016-07-21 08:05:43,609 [myid:] - INFO [main:FourLetterWordMain@62] - connecting to 127.0.0.1 11221
2016-07-21 08:05:43,610 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@58] - Memory used 124233
2016-07-21 08:05:43,611 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@63] - Number of threads 5
2016-07-21 08:05:43,611 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@78] - FINISHED TEST METHOD testDisconnectedAddAuth
2016-07-21 08:05:43,611 [myid:] - INFO [main:ZKTestCase$1@60] - SUCCEEDED testDisconnectedAddAuth
2016-07-21 08:05:43,611 [myid:] - INFO [main:ZKTestCase$1@55] - FINISHED testDisconnectedAddAuth
2016-07-21 08:05:43,613 [myid:] - INFO [main:ZKTestCase$1@50] - STARTING testAcls
2016-07-21 08:05:43,613 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@53] - RUNNING TEST METHOD testAcls
2016-07-21 08:05:43,614 [myid:] - INFO [main:ZooKeeperServer@170] - Created server with tickTime 3000 minSessionTimeout 6000 maxSessionTimeout 60000 datadir /home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/test/tmp/test2000752507502946075.junit.dir/version-2 snapdir /home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/test/tmp/test2000752507502946075.junit.dir/version-2
2016-07-21 08:05:43,635 [myid:] - INFO [main:NettyServerCnxnFactory@365] - binding to port 0.0.0.0/0.0.0.0:11221
2016-07-21 08:05:43,639 [myid:] - INFO [main:ACLTest@98] - starting up the zookeeper server .. waiting
2016-07-21 08:05:43,639 [myid:] - INFO [main:FourLetterWordMain@62] - connecting to 127.0.0.1 11221
2016-07-21 08:05:43,640 [myid:] - INFO [New I/O worker #34:NettyServerCnxn@632] - Processing stat command from /127.0.0.1:35236
2016-07-21 08:05:43,641 [myid:] - INFO [New I/O worker #34:NettyServerCnxn$StatCommand@469] - Stat command output
2016-07-21 08:05:43,641 [myid:] - INFO [main:ZooKeeper@438] - Initiating client connection, connectString=127.0.0.1:11221 sessionTimeout=30000 watcher=org.apache.zookeeper.test.ACLTest@114f6322
2016-07-21 08:05:43,642 [myid:] - INFO [main:ACLTest@102] - starting creating acls
2016-07-21 08:05:43,642 [myid:] - INFO [main-SendThread(127.0.0.1:11221):ClientCnxn$SendThread@1032] - Opening socket connection to server 127.0.0.1/127.0.0.1:11221. Will not attempt to authenticate using SASL (unknown error)
2016-07-21 08:05:43,643 [myid:] - INFO [main-SendThread(127.0.0.1:11221):ClientCnxn$SendThread@876] - Socket connection established to 127.0.0.1/127.0.0.1:11221, initiating session
2016-07-21 08:05:43,644 [myid:] - INFO [New I/O worker #35:ZooKeeperServer@900] - Client attempting to establish new session at /127.0.0.1:35237
2016-07-21 08:05:43,644 [myid:] - INFO [SyncThread:0:FileTxnLog@203] - Creating new log file: log.1
2016-07-21 08:05:43,648 [myid:] - INFO [SyncThread:0:ZooKeeperServer@645] - Established session 0x1560c7d26540000 with negotiated timeout 30000 for client /127.0.0.1:35237
2016-07-21 08:05:43,648 [myid:] - INFO [main-SendThread(127.0.0.1:11221):ClientCnxn$SendThread@1299] - Session establishment complete on server 127.0.0.1/127.0.0.1:11221, sessionid = 0x1560c7d26540000, negotiated timeout = 30000
2016-07-21 08:05:43,648 [myid:] - INFO [main-EventThread:ACLTest@175] - Event:SyncConnected None null
2016-07-21 08:05:43,648 [myid:] - WARN [main-EventThread:ACLTest@182] - startsignal null
2016-07-21 08:05:43,954 [myid:] - INFO [main:NettyServerCnxnFactory@342] - shutdown called 0.0.0.0/0.0.0.0:11221
2016-07-21 08:05:43,955 [myid:] - INFO [main-SendThread(127.0.0.1:11221):ClientCnxn$SendThread@1158] - Unable to read additional data from server sessionid 0x1560c7d26540000, likely server has closed socket, closing socket connection and attempting reconnect
2016-07-21 08:05:43,961 [myid:] - INFO [main:ZooKeeperServer@469] - shutting down
2016-07-21 08:05:43,961 [myid:] - INFO [main:SessionTrackerImpl@225] - Shutting down
2016-07-21 08:05:43,961 [myid:] - INFO [main:PrepRequestProcessor@765] - Shutting down
2016-07-21 08:05:43,962 [myid:] - INFO [main:SyncRequestProcessor@209] - Shutting down
2016-07-21 08:05:43,962 [myid:] - INFO [ProcessThread(sid:0 cport:11221)::PrepRequestProcessor@143] - PrepRequestProcessor exited loop!
2016-07-21 08:05:43,962 [myid:] - INFO [SyncThread:0:SyncRequestProcessor@187] - SyncRequestProcessor exited!
2016-07-21 08:05:43,963 [myid:] - INFO [main:FinalRequestProcessor@402] - shutdown of request processor complete
2016-07-21 08:05:43,963 [myid:] - INFO [main:FourLetterWordMain@62] - connecting to 127.0.0.1 11221
2016-07-21 08:05:43,964 [myid:] - INFO [main:ZooKeeperServer@170] - Created server with tickTime 3000 minSessionTimeout 6000 maxSessionTimeout 60000 datadir /home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/test/tmp/test2000752507502946075.junit.dir/version-2 snapdir /home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/test/tmp/test2000752507502946075.junit.dir/version-2
2016-07-21 08:05:43,972 [myid:] - INFO [main:NettyServerCnxnFactory@365] - binding to port 0.0.0.0/0.0.0.0:11221
2016-07-21 08:05:44,002 [myid:] - INFO [main:FourLetterWordMain@62] - connecting to 127.0.0.1 11221
2016-07-21 08:05:44,004 [myid:] - INFO [New I/O worker #67:NettyServerCnxn@632] - Processing stat command from /127.0.0.1:35240
2016-07-21 08:05:44,004 [myid:] - INFO [New I/O worker #67:NettyServerCnxn$StatCommand@469] - Stat command output
2016-07-21 08:05:44,055 [myid:] - INFO [main-EventThread:ACLTest@175] - Event:Disconnected None null
2016-07-21 08:05:45,000 [myid:] - INFO [SessionTracker:SessionTrackerImpl@162] - SessionTrackerImpl exited loop!
2016-07-21 08:05:45,000 [myid:] - INFO [SessionTracker:SessionTrackerImpl@162] - SessionTrackerImpl exited loop!
2016-07-21 08:05:45,784 [myid:] - INFO [main-SendThread(127.0.0.1:11221):ClientCnxn$SendThread@1032] - Opening socket connection to server 127.0.0.1/127.0.0.1:11221. Will not attempt to authenticate using SASL (unknown error)
2016-07-21 08:05:45,784 [myid:] - INFO [main-SendThread(127.0.0.1:11221):ClientCnxn$SendThread@876] - Socket connection established to 127.0.0.1/127.0.0.1:11221, initiating session
2016-07-21 08:05:45,785 [myid:] - INFO [New I/O worker #68:ZooKeeperServer@893] - Client attempting to renew session 0x1560c7d26540000 at /127.0.0.1:35246
2016-07-21 08:05:45,786 [myid:] - INFO [New I/O worker #68:ZooKeeperServer@645] - Established session 0x1560c7d26540000 with negotiated timeout 30000 for client /127.0.0.1:35246
2016-07-21 08:05:45,786 [myid:] - INFO [main-SendThread(127.0.0.1:11221):ClientCnxn$SendThread@1299] - Session establishment complete on server 127.0.0.1/127.0.0.1:11221, sessionid = 0x1560c7d26540000, negotiated timeout = 30000
2016-07-21 08:05:45,786 [myid:] - INFO [main-EventThread:ACLTest@175] - Event:SyncConnected None null
2016-07-21 08:05:45,786 [myid:] - INFO [main-EventThread:ACLTest@179] - startsignal.countDown()
2016-07-21 08:05:45,788 [myid:] - INFO [SyncThread:0:FileTxnLog@203] - Creating new log file: log.ca
2016-07-21 08:05:45,795 [myid:] - INFO [ProcessThread(sid:0 cport:11221)::PrepRequestProcessor@487] - Processed session termination for sessionid: 0x1560c7d26540000
2016-07-21 08:05:45,795 [myid:] - WARN [New I/O worker #68:NettyServerCnxnFactory$CnxnChannelHandler@111] - Exception caught [id: 0x0ae869d5, /127.0.0.1:35246 :> /127.0.0.1:11221] EXCEPTION: java.nio.channels.ClosedChannelException
java.nio.channels.ClosedChannelException
at sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:270)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:479)
at org.jboss.netty.channel.socket.nio.SocketSendBufferPool$UnpooledSendBuffer.transferTo(SocketSendBufferPool.java:203)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:201)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromTaskLoop(AbstractNioWorker.java:151)
at org.jboss.netty.channel.socket.nio.AbstractNioChannel$WriteTask.run(AbstractNioChannel.java:315)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:391)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:315)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2016-07-21 08:05:45,896 [myid:] - INFO [main:ZooKeeper@684] - Session: 0x1560c7d26540000 closed
2016-07-21 08:05:45,896 [myid:] - INFO [main:NettyServerCnxnFactory@342] - shutdown called 0.0.0.0/0.0.0.0:11221
2016-07-21 08:05:45,896 [myid:] - INFO [main-EventThread:ClientCnxn$EventThread@519] - EventThread shut down for session: 0x1560c7d26540000
2016-07-21 08:05:45,901 [myid:] - INFO [main:ZooKeeperServer@469] - shutting down
2016-07-21 08:05:45,901 [myid:] - INFO [main:SessionTrackerImpl@225] - Shutting down
2016-07-21 08:05:45,901 [myid:] - INFO [main:PrepRequestProcessor@765] - Shutting down
2016-07-21 08:05:45,902 [myid:] - INFO [ProcessThread(sid:0 cport:11221)::PrepRequestProcessor@143] - PrepRequestProcessor exited loop!
2016-07-21 08:05:45,902 [myid:] - INFO [main:SyncRequestProcessor@209] - Shutting down
2016-07-21 08:05:45,902 [myid:] - INFO [SyncThread:0:SyncRequestProcessor@187] - SyncRequestProcessor exited!
2016-07-21 08:05:45,902 [myid:] - INFO [main:FinalRequestProcessor@402] - shutdown of request processor complete
2016-07-21 08:05:45,903 [myid:] - INFO [main:FourLetterWordMain@62] - connecting to 127.0.0.1 11221
2016-07-21 08:05:45,904 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@58] - Memory used 81805
2016-07-21 08:05:45,904 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@63] - Number of threads 7
2016-07-21 08:05:45,904 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@78] - FINISHED TEST METHOD testAcls
2016-07-21 08:05:45,904 [myid:] - INFO [main:ZKTestCase$1@60] - SUCCEEDED testAcls
2016-07-21 08:05:45,905 [myid:] - INFO [main:ZKTestCase$1@55] - FINISHED testAcls
2016-07-21 08:05:45,914 [myid:] - INFO [main:PortAssignment@32] - assigning port 11222
2016-07-21 08:05:45,915 [myid:] - INFO [main:ZKTestCase$1@50] - STARTING testAsyncCreate
2016-07-21 08:05:45,916 [myid:] - INFO [main:ClientBase@425] - Initial fdcount is: 33
2016-07-21 08:05:46,035 [myid:] - INFO [main:ClientBase@443] - STARTING server
2016-07-21 08:05:46,035 [myid:] - INFO [main:ClientBase@364] - CREATING server instance 127.0.0.1:11222
2016-07-21 08:05:46,042 [myid:] - INFO [main:ClientBase@339] - STARTING server instance 127.0.0.1:11222
2016-07-21 08:05:46,043 [myid:] - INFO [main:ZooKeeperServer@170] - Created server with tickTime 3000 minSessionTimeout 6000 maxSessionTimeout 60000 datadir /home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/test/tmp/test4160740823067989579.junit.dir/version-2 snapdir /home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/test/tmp/test4160740823067989579.junit.dir/version-2
2016-07-21 08:05:46,043 [myid:] - INFO [main:NettyServerCnxnFactory@365] - binding to port 0.0.0.0/0.0.0.0:11222
2016-07-21 08:05:46,046 [myid:] - INFO [main:FourLetterWordMain@62] - connecting to 127.0.0.1 11222
2016-07-21 08:05:46,047 [myid:] - INFO [New I/O worker #100:NettyServerCnxn@632] - Processing stat command from /127.0.0.1:49831
2016-07-21 08:05:46,047 [myid:] - INFO [New I/O worker #100:NettyServerCnxn$StatCommand@469] - Stat command output
2016-07-21 08:05:46,048 [myid:] - INFO [main:JMXEnv@229] - ensureParent:[InMemoryDataTree, StandaloneServer_port]
2016-07-21 08:05:46,053 [myid:] - INFO [main:JMXEnv@246] - expect:InMemoryDataTree
2016-07-21 08:05:46,053 [myid:] - INFO [main:JMXEnv@250] - found:InMemoryDataTree org.apache.ZooKeeperService:name0=StandaloneServer_port11222,name1=InMemoryDataTree
2016-07-21 08:05:46,054 [myid:] - INFO [main:JMXEnv@246] - expect:StandaloneServer_port
2016-07-21 08:05:46,054 [myid:] - INFO [main:JMXEnv@250] - found:StandaloneServer_port org.apache.ZooKeeperService:name0=StandaloneServer_port11222
2016-07-21 08:05:46,054 [myid:] - INFO [main:ClientBase@439] - Client test setup finished
2016-07-21 08:05:46,054 [myid:] - INFO [main:AsyncOpsTest@47] - Creating client testAsyncCreate
2016-07-21 08:05:46,055 [myid:] - INFO [main:ZooKeeper@438] - Initiating client connection, connectString=127.0.0.1:11222 sessionTimeout=30000 watcher=org.apache.zookeeper.test.ClientBase$CountdownWatcher@42701c57
2016-07-21 08:05:46,056 [myid:] - INFO [main-SendThread(127.0.0.1:11222):ClientCnxn$SendThread@1032] - Opening socket connection to server 127.0.0.1/127.0.0.1:11222. Will not attempt to authenticate using SASL (unknown error)
2016-07-21 08:05:46,056 [myid:] - INFO [main-SendThread(127.0.0.1:11222):ClientCnxn$SendThread@876] - Socket connection established to 127.0.0.1/127.0.0.1:11222, initiating session
2016-07-21 08:05:46,057 [myid:] - INFO [New I/O worker #101:ZooKeeperServer@900] - Client attempting to establish new session at /127.0.0.1:49832
2016-07-21 08:05:46,057 [myid:] - INFO [SyncThread:0:FileTxnLog@203] - Creating new log file: log.1
2016-07-21 08:05:46,060 [myid:] - INFO [SyncThread:0:ZooKeeperServer@645] - Established session 0x1560c7d2fbb0000 with negotiated timeout 30000 for client /127.0.0.1:49832
2016-07-21 08:05:46,060 [myid:] - INFO [main-SendThread(127.0.0.1:11222):ClientCnxn$SendThread@1299] - Session establishment complete on server 127.0.0.1/127.0.0.1:11222, sessionid = 0x1560c7d2fbb0000, negotiated timeout = 30000
2016-07-21 08:05:46,062 [myid:] - INFO [main:JMXEnv@117] - expect:0x1560c7d2fbb0000
2016-07-21 08:05:46,062 [myid:] - INFO [main:JMXEnv@120] - found:0x1560c7d2fbb0000 org.apache.ZooKeeperService:name0=StandaloneServer_port11222,name1=Connections,name2=127.0.0.1,name3=0x1560c7d2fbb0000
2016-07-21 08:05:46,063 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@53] - RUNNING TEST METHOD testAsyncCreate
2016-07-21 08:05:46,063 [myid:] - INFO [New I/O worker #101:ZooKeeperServer@924] - got auth packet /127.0.0.1:49832
2016-07-21 08:05:46,063 [myid:] - INFO [New I/O worker #101:ZooKeeperServer@958] - auth success /127.0.0.1:49832
2016-07-21 08:05:46,069 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@58] - Memory used 40629
2016-07-21 08:05:46,069 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@63] - Number of threads 51
2016-07-21 08:05:46,069 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@78] - FINISHED TEST METHOD testAsyncCreate
2016-07-21 08:05:46,070 [myid:] - INFO [ProcessThread(sid:0 cport:11222)::PrepRequestProcessor@487] - Processed session termination for sessionid: 0x1560c7d2fbb0000
2016-07-21 08:05:46,070 [myid:] - WARN [New I/O worker #101:NettyServerCnxnFactory$CnxnChannelHandler@111] - Exception caught [id: 0xe25524be, /127.0.0.1:49832 :> /127.0.0.1:11222] EXCEPTION: java.nio.channels.ClosedChannelException
java.nio.channels.ClosedChannelException
at sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:270)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:479)
at org.jboss.netty.channel.socket.nio.SocketSendBufferPool$UnpooledSendBuffer.transferTo(SocketSendBufferPool.java:203)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:201)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromTaskLoop(AbstractNioWorker.java:151)
at org.jboss.netty.channel.socket.nio.AbstractNioChannel$WriteTask.run(AbstractNioChannel.java:315)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:391)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:315)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2016-07-21 08:05:46,171 [myid:] - INFO [main:ZooKeeper@684] - Session: 0x1560c7d2fbb0000 closed
2016-07-21 08:05:46,171 [myid:] - INFO [main:ClientBase@520] - tearDown starting
2016-07-21 08:05:46,171 [myid:] - INFO [main-EventThread:ClientCnxn$EventThread@519] - EventThread shut down for session: 0x1560c7d2fbb0000
2016-07-21 08:05:46,171 [myid:] - INFO [main:ClientBase@490] - STOPPING server
2016-07-21 08:05:46,172 [myid:] - INFO [main:NettyServerCnxnFactory@342] - shutdown called 0.0.0.0/0.0.0.0:11222
2016-07-21 08:05:46,183 [myid:] - INFO [main:ZooKeeperServer@469] - shutting down
2016-07-21 08:05:46,183 [myid:] - INFO [main:SessionTrackerImpl@225] - Shutting down
2016-07-21 08:05:46,183 [myid:] - INFO [main:PrepRequestProcessor@765] - Shutting down
2016-07-21 08:05:46,184 [myid:] - INFO [main:SyncRequestProcessor@209] - Shutting down
2016-07-21 08:05:46,184 [myid:] - INFO [ProcessThread(sid:0 cport:11222)::PrepRequestProcessor@143] - PrepRequestProcessor exited loop!
2016-07-21 08:05:46,184 [myid:] - INFO [SyncThread:0:SyncRequestProcessor@187] - SyncRequestProcessor exited!
2016-07-21 08:05:46,185 [myid:] - INFO [main:FinalRequestProcessor@402] - shutdown of request processor complete
2016-07-21 08:05:46,185 [myid:] - INFO [main:FourLetterWordMain@62] - connecting to 127.0.0.1 11222
2016-07-21 08:05:46,186 [myid:] - INFO [main:JMXEnv@146] - ensureOnly:[]
2016-07-21 08:05:46,189 [myid:] - INFO [main:ClientBase@545] - fdcount after test is: 34 at start it was 33
2016-07-21 08:05:46,190 [myid:] - INFO [main:ClientBase@547] - sleeping for 20 secs
2016-07-21 08:05:46,190 [myid:] - INFO [main:AsyncOpsTest@60] - Test clients shutting down
2016-07-21 08:05:46,191 [myid:] - INFO [main:ZKTestCase$1@60] - SUCCEEDED testAsyncCreate
2016-07-21 08:05:46,191 [myid:] - INFO [main:ZKTestCase$1@55] - FINISHED testAsyncCreate
2016-07-21 08:05:46,192 [myid:] - INFO [main:PortAssignment@32] - assigning port 11223
2016-07-21 08:05:46,193 [myid:] - INFO [main:ZKTestCase$1@50] - STARTING testAsyncCreateThree
2016-07-21 08:05:46,193 [myid:] - INFO [main:ClientBase@425] - Initial fdcount is: 34
2016-07-21 08:05:46,204 [myid:] - INFO [main:ClientBase@443] - STARTING server
2016-07-21 08:05:46,204 [myid:] - INFO [main:ClientBase@364] - CREATING server instance 127.0.0.1:11223
2016-07-21 08:05:46,212 [myid:] - INFO [main:ClientBase@339] - STARTING server instance 127.0.0.1:11223
2016-07-21 08:05:46,212 [myid:] - INFO [main:ZooKeeperServer@170] - Created server with tickTime 3000 minSessionTimeout 6000 maxSessionTimeout 60000 datadir /home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/test/tmp/test8798982101480729726.junit.dir/version-2 snapdir /home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/test/tmp/test8798982101480729726.junit.dir/version-2
2016-07-21 08:05:46,212 [myid:] - INFO [main:NettyServerCnxnFactory@365] - binding to port 0.0.0.0/0.0.0.0:11223
2016-07-21 08:05:46,214 [myid:] - INFO [main:FourLetterWordMain@62] - connecting to 127.0.0.1 11223
2016-07-21 08:05:46,215 [myid:] - INFO [New I/O worker #133:NettyServerCnxn@632] - Processing stat command from /127.0.0.1:38741
2016-07-21 08:05:46,215 [myid:] - INFO [New I/O worker #133:NettyServerCnxn$StatCommand@469] - Stat command output
2016-07-21 08:05:46,216 [myid:] - INFO [main:JMXEnv@229] - ensureParent:[InMemoryDataTree, StandaloneServer_port]
2016-07-21 08:05:46,218 [myid:] - INFO [main:JMXEnv@246] - expect:InMemoryDataTree
2016-07-21 08:05:46,218 [myid:] - INFO [main:JMXEnv@250] - found:InMemoryDataTree org.apache.ZooKeeperService:name0=StandaloneServer_port11223,name1=InMemoryDataTree
2016-07-21 08:05:46,218 [myid:] - INFO [main:JMXEnv@246] - expect:StandaloneServer_port
2016-07-21 08:05:46,218 [myid:] - INFO [main:JMXEnv@250] - found:StandaloneServer_port org.apache.ZooKeeperService:name0=StandaloneServer_port11223
2016-07-21 08:05:46,218 [myid:] - INFO [main:ClientBase@439] - Client test setup finished
2016-07-21 08:05:46,219 [myid:] - INFO [main:AsyncOpsTest@47] - Creating client testAsyncCreateThree
2016-07-21 08:05:46,219 [myid:] - INFO [main:ZooKeeper@438] - Initiating client connection, connectString=127.0.0.1:11223 sessionTimeout=30000 watcher=org.apache.zookeeper.test.ClientBase$CountdownWatcher@41327667
2016-07-21 08:05:46,220 [myid:] - INFO [main-SendThread(127.0.0.1:11223):ClientCnxn$SendThread@1032] - Opening socket connection to server 127.0.0.1/127.0.0.1:11223. Will not attempt to authenticate using SASL (unknown error)
2016-07-21 08:05:46,220 [myid:] - INFO [main-SendThread(127.0.0.1:11223):ClientCnxn$SendThread@876] - Socket connection established to 127.0.0.1/127.0.0.1:11223, initiating session
2016-07-21 08:05:46,221 [myid:] - INFO [New I/O worker #134:ZooKeeperServer@900] - Client attempting to establish new session at /127.0.0.1:38742
2016-07-21 08:05:46,221 [myid:] - INFO [SyncThread:0:FileTxnLog@203] - Creating new log file: log.1
2016-07-21 08:05:46,223 [myid:] - INFO [SyncThread:0:ZooKeeperServer@645] - Established session 0x1560c7d30650000 with negotiated timeout 30000 for client /127.0.0.1:38742
2016-07-21 08:05:46,224 [myid:] - INFO [main-SendThread(127.0.0.1:11223):ClientCnxn$SendThread@1299] - Session establishment complete on server 127.0.0.1/127.0.0.1:11223, sessionid = 0x1560c7d30650000, negotiated timeout = 30000
2016-07-21 08:05:46,226 [myid:] - INFO [main:JMXEnv@117] - expect:0x1560c7d30650000
2016-07-21 08:05:46,226 [myid:] - INFO [main:JMXEnv@120] - found:0x1560c7d30650000 org.apache.ZooKeeperService:name0=StandaloneServer_port11223,name1=Connections,name2=127.0.0.1,name3=0x1560c7d30650000
2016-07-21 08:05:46,226 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@53] - RUNNING TEST METHOD testAsyncCreateThree
2016-07-21 08:05:46,227 [myid:] - INFO [New I/O worker #134:ZooKeeperServer@924] - got auth packet /127.0.0.1:38742
2016-07-21 08:05:46,227 [myid:] - INFO [New I/O worker #134:ZooKeeperServer@958] - auth success /127.0.0.1:38742
2016-07-21 08:05:46,230 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@58] - Memory used 77876
2016-07-21 08:05:46,230 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@63] - Number of threads 52
2016-07-21 08:05:46,230 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@78] - FINISHED TEST METHOD testAsyncCreateThree
2016-07-21 08:05:46,231 [myid:] - INFO [ProcessThread(sid:0 cport:11223)::PrepRequestProcessor@487] - Processed session termination for sessionid: 0x1560c7d30650000
2016-07-21 08:05:46,232 [myid:] - WARN [New I/O worker #134:NettyServerCnxnFactory$CnxnChannelHandler@111] - Exception caught [id: 0x84929b58, /127.0.0.1:38742 :> /127.0.0.1:11223] EXCEPTION: java.nio.channels.ClosedChannelException
java.nio.channels.ClosedChannelException
at sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:270)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:479)
at org.jboss.netty.channel.socket.nio.SocketSendBufferPool$UnpooledSendBuffer.transferTo(SocketSendBufferPool.java:203)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:201)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromTaskLoop(AbstractNioWorker.java:151)
at org.jboss.netty.channel.socket.nio.AbstractNioChannel$WriteTask.run(AbstractNioChannel.java:315)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:391)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:315)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2016-07-21 08:05:46,332 [myid:] - INFO [main:ZooKeeper@684] - Session: 0x1560c7d30650000 closed
2016-07-21 08:05:46,333 [myid:] - INFO [main:ClientBase@520] - tearDown starting
2016-07-21 08:05:46,333 [myid:] - INFO [main:ClientBase@490] - STOPPING server
2016-07-21 08:05:46,332 [myid:] - INFO [main-EventThread:ClientCnxn$EventThread@519] - EventThread shut down for session: 0x1560c7d30650000
2016-07-21 08:05:46,333 [myid:] - INFO [main:NettyServerCnxnFactory@342] - shutdown called 0.0.0.0/0.0.0.0:11223
2016-07-21 08:05:46,340 [myid:] - INFO [main:ZooKeeperServer@469] - shutting down
2016-07-21 08:05:46,341 [myid:] - INFO [main:SessionTrackerImpl@225] - Shutting down
2016-07-21 08:05:46,341 [myid:] - INFO [main:PrepRequestProcessor@765] - Shutting down
2016-07-21 08:05:46,341 [myid:] - INFO [main:SyncRequestProcessor@209] - Shutting down
2016-07-21 08:05:46,341 [myid:] - INFO [ProcessThread(sid:0 cport:11223)::PrepRequestProcessor@143] - PrepRequestProcessor exited loop!
2016-07-21 08:05:46,342 [myid:] - INFO [SyncThread:0:SyncRequestProcessor@187] - SyncRequestProcessor exited!
2016-07-21 08:05:46,342 [myid:] - INFO [main:FinalRequestProcessor@402] - shutdown of request processor complete
2016-07-21 08:05:46,343 [myid:] - INFO [main:FourLetterWordMain@62] - connecting to 127.0.0.1 11223
2016-07-21 08:05:46,343 [myid:] - INFO [main:JMXEnv@146] - ensureOnly:[]
2016-07-21 08:05:46,346 [myid:] - INFO [main:ClientBase@545] - fdcount after test is: 34 at start it was 34
2016-07-21 08:05:46,346 [myid:] - INFO [main:AsyncOpsTest@60] - Test clients shutting down
2016-07-21 08:05:46,347 [myid:] - INFO [main:ZKTestCase$1@60] - SUCCEEDED testAsyncCreateThree
2016-07-21 08:05:46,347 [myid:] - INFO [main:ZKTestCase$1@55] - FINISHED testAsyncCreateThree
2016-07-21 08:05:46,347 [myid:] - INFO [main:PortAssignment@32] - assigning port 11224
2016-07-21 08:05:46,348 [myid:] - INFO [main:ZKTestCase$1@50] - STARTING testAsyncCreateFailure_NodeExists
2016-07-21 08:05:46,348 [myid:] - INFO [main:ClientBase@425] - Initial fdcount is: 34
2016-07-21 08:05:46,355 [myid:] - INFO [main:ClientBase@443] - STARTING server
2016-07-21 08:05:46,355 [myid:] - INFO [main:ClientBase@364] - CREATING server instance 127.0.0.1:11224
2016-07-21 08:05:46,362 [myid:] - INFO [main:ClientBase@339] - STARTING server instance 127.0.0.1:11224
2016-07-21 08:05:46,363 [myid:] - INFO [main:ZooKeeperServer@170] - Created server with tickTime 3000 minSessionTimeout 6000 maxSessionTimeout 60000 datadir /home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/test/tmp/test937004609623311539.junit.dir/version-2 snapdir /home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/test/tmp/test937004609623311539.junit.dir/version-2
2016-07-21 08:05:46,363 [myid:] - INFO [main:NettyServerCnxnFactory@365] - binding to port 0.0.0.0/0.0.0.0:11224
2016-07-21 08:05:46,364 [myid:] - INFO [main:FourLetterWordMain@62] - connecting to 127.0.0.1 11224
2016-07-21 08:05:46,365 [myid:] - INFO [New I/O worker #166:NettyServerCnxn@632] - Processing stat command from /127.0.0.1:54116
2016-07-21 08:05:46,366 [myid:] - INFO [New I/O worker #166:NettyServerCnxn$StatCommand@469] - Stat command output
2016-07-21 08:05:46,366 [myid:] - INFO [main:JMXEnv@229] - ensureParent:[InMemoryDataTree, StandaloneServer_port]
2016-07-21 08:05:46,368 [myid:] - INFO [main:JMXEnv@246] - expect:InMemoryDataTree
2016-07-21 08:05:46,368 [myid:] - INFO [main:JMXEnv@250] - found:InMemoryDataTree org.apache.ZooKeeperService:name0=StandaloneServer_port11224,name1=InMemoryDataTree
2016-07-21 08:05:46,368 [myid:] - INFO [main:JMXEnv@246] - expect:StandaloneServer_port
2016-07-21 08:05:46,369 [myid:] - INFO [main:JMXEnv@250] - found:StandaloneServer_port org.apache.ZooKeeperService:name0=StandaloneServer_port11224
2016-07-21 08:05:46,369 [myid:] - INFO [main:ClientBase@439] - Client test setup finished
2016-07-21 08:05:46,369 [myid:] - INFO [main:AsyncOpsTest@47] - Creating client testAsyncCreateFailure_NodeExists
2016-07-21 08:05:46,369 [myid:] - INFO [main:ZooKeeper@438] - Initiating client connection, connectString=127.0.0.1:11224 sessionTimeout=30000 watcher=org.apache.zookeeper.test.ClientBase$CountdownWatcher@78f394a2
2016-07-21 08:05:46,370 [myid:] - INFO [main-SendThread(127.0.0.1:11224):ClientCnxn$SendThread@1032] - Opening socket connection to server 127.0.0.1/127.0.0.1:11224. Will not attempt to authenticate using SASL (unknown error)
2016-07-21 08:05:46,371 [myid:] - INFO [main-SendThread(127.0.0.1:11224):ClientCnxn$SendThread@876] - Socket connection established to 127.0.0.1/127.0.0.1:11224, initiating session
2016-07-21 08:05:46,371 [myid:] - INFO [New I/O worker #167:ZooKeeperServer@900] - Client attempting to establish new session at /127.0.0.1:54117
2016-07-21 08:05:46,372 [myid:] - INFO [SyncThread:0:FileTxnLog@203] - Creating new log file: log.1
2016-07-21 08:05:46,374 [myid:] - INFO [SyncThread:0:ZooKeeperServer@645] - Established session 0x1560c7d30fb0000 with negotiated timeout 30000 for client /127.0.0.1:54117
2016-07-21 08:05:46,374 [myid:] - INFO [main-SendThread(127.0.0.1:11224):ClientCnxn$SendThread@1299] - Session establishment complete on server 127.0.0.1/127.0.0.1:11224, sessionid = 0x1560c7d30fb0000, negotiated timeout = 30000
2016-07-21 08:05:46,376 [myid:] - INFO [main:JMXEnv@117] - expect:0x1560c7d30fb0000
2016-07-21 08:05:46,377 [myid:] - INFO [main:JMXEnv@120] - found:0x1560c7d30fb0000 org.apache.ZooKeeperService:name0=StandaloneServer_port11224,name1=Connections,name2=127.0.0.1,name3=0x1560c7d30fb0000
2016-07-21 08:05:46,377 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@53] - RUNNING TEST METHOD testAsyncCreateFailure_NodeExists
2016-07-21 08:05:46,377 [myid:] - INFO [New I/O worker #167:ZooKeeperServer@924] - got auth packet /127.0.0.1:54117
2016-07-21 08:05:46,378 [myid:] - INFO [New I/O worker #167:ZooKeeperServer@958] - auth success /127.0.0.1:54117
2016-07-21 08:05:46,382 [myid:] - INFO [ProcessThread(sid:0 cport:11224)::PrepRequestProcessor@649] - Got user-level KeeperException when processing sessionid:0x1560c7d30fb0000 type:create cxid:0x2 zxid:0x3 txntype:-1 reqpath:n/a Error Path:/foo Error:KeeperErrorCode = NodeExists for /foo
2016-07-21 08:05:46,382 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@58] - Memory used 115887
2016-07-21 08:05:46,383 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@63] - Number of threads 53
2016-07-21 08:05:46,383 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@78] - FINISHED TEST METHOD testAsyncCreateFailure_NodeExists
2016-07-21 08:05:46,384 [myid:] - INFO [ProcessThread(sid:0 cport:11224)::PrepRequestProcessor@487] - Processed session termination for sessionid: 0x1560c7d30fb0000
2016-07-21 08:05:46,385 [myid:] - WARN [New I/O worker #167:NettyServerCnxnFactory$CnxnChannelHandler@111] - Exception caught [id: 0x08491bfc, /127.0.0.1:54117 :> /127.0.0.1:11224] EXCEPTION: java.nio.channels.ClosedChannelException
java.nio.channels.ClosedChannelException
at sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:270)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:479)
at org.jboss.netty.channel.socket.nio.SocketSendBufferPool$UnpooledSendBuffer.transferTo(SocketSendBufferPool.java:203)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:201)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromTaskLoop(AbstractNioWorker.java:151)
at org.jboss.netty.channel.socket.nio.AbstractNioChannel$WriteTask.run(AbstractNioChannel.java:315)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:391)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:315)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2016-07-21 08:05:46,485 [myid:] - INFO [main:ZooKeeper@684] - Session: 0x1560c7d30fb0000 closed
2016-07-21 08:05:46,485 [myid:] - INFO [main:ClientBase@520] - tearDown starting
2016-07-21 08:05:46,485 [myid:] - INFO [main:ClientBase@490] - STOPPING server
2016-07-21 08:05:46,486 [myid:] - INFO [main:NettyServerCnxnFactory@342] - shutdown called 0.0.0.0/0.0.0.0:11224
2016-07-21 08:05:46,485 [myid:] - INFO [main-EventThread:ClientCnxn$EventThread@519] - EventThread shut down for session: 0x1560c7d30fb0000
2016-07-21 08:05:46,495 [myid:] - INFO [main:ZooKeeperServer@469] - shutting down
2016-07-21 08:05:46,495 [myid:] - INFO [main:SessionTrackerImpl@225] - Shutting down
2016-07-21 08:05:46,495 [myid:] - INFO [main:PrepRequestProcessor@765] - Shutting down
2016-07-21 08:05:46,496 [myid:] - INFO [main:SyncRequestProcessor@209] - Shutting down
2016-07-21 08:05:46,496 [myid:] - INFO [ProcessThread(sid:0 cport:11224)::PrepRequestProcessor@143] - PrepRequestProcessor exited loop!
2016-07-21 08:05:46,496 [myid:] - INFO [SyncThread:0:SyncRequestProcessor@187] - SyncRequestProcessor exited!
2016-07-21 08:05:46,496 [myid:] - INFO [main:FinalRequestProcessor@402] - shutdown of request processor complete
2016-07-21 08:05:46,498 [myid:] - INFO [main:FourLetterWordMain@62] - connecting to 127.0.0.1 11224
2016-07-21 08:05:46,499 [myid:] - INFO [main:JMXEnv@146] - ensureOnly:[]
2016-07-21 08:05:46,502 [myid:] - INFO [main:ClientBase@545] - fdcount after test is: 34 at start it was 34
2016-07-21 08:05:46,502 [myid:] - INFO [main:AsyncOpsTest@60] - Test clients shutting down
2016-07-21 08:05:46,502 [myid:] - INFO [main:ZKTestCase$1@60] - SUCCEEDED testAsyncCreateFailure_NodeExists
2016-07-21 08:05:46,502 [myid:] - INFO [main:ZKTestCase$1@55] - FINISHED testAsyncCreateFailure_NodeExists
2016-07-21 08:05:46,503 [myid:] - INFO [main:PortAssignment@32] - assigning port 11225
2016-07-21 08:05:46,503 [myid:] - INFO [main:ZKTestCase$1@50] - STARTING testAsyncCreateFailure_NoNode
2016-07-21 08:05:46,504 [myid:] - INFO [main:ClientBase@425] - Initial fdcount is: 34
2016-07-21 08:05:46,514 [myid:] - INFO [main:ClientBase@443] - STARTING server
2016-07-21 08:05:46,514 [myid:] - INFO [main:ClientBase@364] - CREATING server instance 127.0.0.1:11225
2016-07-21 08:05:46,525 [myid:] - INFO [main:ClientBase@339] - STARTING server instance 127.0.0.1:11225
2016-07-21 08:05:46,525 [myid:] - INFO [main:ZooKeeperServer@170] - Created server with tickTime 3000 minSessionTimeout 6000 maxSessionTimeout 60000 datadir /home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/test/tmp/test5380464042991622148.junit.dir/version-2 snapdir /home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/test/tmp/test5380464042991622148.junit.dir/version-2
2016-07-21 08:05:46,525 [myid:] - INFO [main:NettyServerCnxnFactory@365] - bindin
...[truncated 880056 chars]...
ocessor exited loop!
2016-07-21 08:08:01,738 [myid:] - INFO [SyncThread:0:SyncRequestProcessor@187] - SyncRequestProcessor exited!
2016-07-21 08:08:01,739 [myid:] - INFO [main:FinalRequestProcessor@402] - shutdown of request processor complete
2016-07-21 08:08:01,740 [myid:] - INFO [main:FourLetterWordMain@62] - connecting to 127.0.0.1 11289
2016-07-21 08:08:01,740 [myid:] - INFO [main:JMXEnv@146] - ensureOnly:[]
2016-07-21 08:08:02,332 [myid:] - INFO [main:ClientBase@443] - STARTING server
2016-07-21 08:08:02,332 [myid:] - INFO [main:ClientBase@364] - CREATING server instance 127.0.0.1:11289
2016-07-21 08:08:02,340 [myid:] - INFO [main:ClientBase@339] - STARTING server instance 127.0.0.1:11289
2016-07-21 08:08:02,341 [myid:] - INFO [main:ZooKeeperServer@170] - Created server with tickTime 3000 minSessionTimeout 6000 maxSessionTimeout 60000 datadir /home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/test/tmp/test4028127123136724556.junit.dir/version-2 snapdir /home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/test/tmp/test4028127123136724556.junit.dir/version-2
2016-07-21 08:08:02,341 [myid:] - INFO [main:NettyServerCnxnFactory@365] - binding to port 0.0.0.0/0.0.0.0:11289
2016-07-21 08:08:02,344 [myid:] - INFO [main:FourLetterWordMain@62] - connecting to 127.0.0.1 11289
2016-07-21 08:08:02,345 [myid:] - INFO [New I/O worker #2707:NettyServerCnxn@632] - Processing stat command from /127.0.0.1:36112
2016-07-21 08:08:02,345 [myid:] - INFO [New I/O worker #2707:NettyServerCnxn$StatCommand@469] - Stat command output
2016-07-21 08:08:02,345 [myid:] - INFO [main:JMXEnv@229] - ensureParent:[InMemoryDataTree, StandaloneServer_port]
2016-07-21 08:08:02,346 [myid:] - INFO [main:JMXEnv@246] - expect:InMemoryDataTree
2016-07-21 08:08:02,347 [myid:] - INFO [main:JMXEnv@250] - found:InMemoryDataTree org.apache.ZooKeeperService:name0=StandaloneServer_port11289,name1=InMemoryDataTree
2016-07-21 08:08:02,347 [myid:] - INFO [main:JMXEnv@246] - expect:StandaloneServer_port
2016-07-21 08:08:02,347 [myid:] - INFO [main:JMXEnv@250] - found:StandaloneServer_port org.apache.ZooKeeperService:name0=StandaloneServer_port11289
2016-07-21 08:08:03,000 [myid:] - INFO [SessionTracker:SessionTrackerImpl@162] - SessionTrackerImpl exited loop!
2016-07-21 08:08:03,274 [myid:] - INFO [main-SendThread(127.0.0.1:11289):ClientCnxn$SendThread@1032] - Opening socket connection to server 127.0.0.1/127.0.0.1:11289. Will not attempt to authenticate using SASL (unknown error)
2016-07-21 08:08:03,274 [myid:] - INFO [main-SendThread(127.0.0.1:11289):ClientCnxn$SendThread@876] - Socket connection established to 127.0.0.1/127.0.0.1:11289, initiating session
2016-07-21 08:08:03,275 [myid:] - INFO [New I/O worker #2708:ZooKeeperServer@893] - Client attempting to renew session 0x1560c7f3a630000 at /127.0.0.1:36116
2016-07-21 08:08:03,276 [myid:] - INFO [New I/O worker #2708:ZooKeeperServer@645] - Established session 0x1560c7f3a630000 with negotiated timeout 6000 for client /127.0.0.1:36116
2016-07-21 08:08:03,276 [myid:] - INFO [main-SendThread(127.0.0.1:11289):ClientCnxn$SendThread@1299] - Session establishment complete on server 127.0.0.1/127.0.0.1:11289, sessionid = 0x1560c7f3a630000, negotiated timeout = 6000
2016-07-21 08:08:03,281 [myid:] - INFO [SyncThread:0:FileTxnLog@203] - Creating new log file: log.6
2016-07-21 08:08:03,283 [myid:] - INFO [main:ClientBase@490] - STOPPING server
2016-07-21 08:08:03,283 [myid:] - INFO [main:NettyServerCnxnFactory@342] - shutdown called 0.0.0.0/0.0.0.0:11289
2016-07-21 08:08:03,284 [myid:] - INFO [main-SendThread(127.0.0.1:11289):ClientCnxn$SendThread@1158] - Unable to read additional data from server sessionid 0x1560c7f3a630000, likely server has closed socket, closing socket connection and attempting reconnect
2016-07-21 08:08:03,293 [myid:] - INFO [main:ZooKeeperServer@469] - shutting down
2016-07-21 08:08:03,295 [myid:] - INFO [main:SessionTrackerImpl@225] - Shutting down
2016-07-21 08:08:03,296 [myid:] - INFO [main:PrepRequestProcessor@765] - Shutting down
2016-07-21 08:08:03,296 [myid:] - INFO [main:SyncRequestProcessor@209] - Shutting down
2016-07-21 08:08:03,296 [myid:] - INFO [ProcessThread(sid:0 cport:11289)::PrepRequestProcessor@143] - PrepRequestProcessor exited loop!
2016-07-21 08:08:03,296 [myid:] - INFO [SyncThread:0:SyncRequestProcessor@187] - SyncRequestProcessor exited!
2016-07-21 08:08:03,297 [myid:] - INFO [main:FinalRequestProcessor@402] - shutdown of request processor complete
2016-07-21 08:08:03,298 [myid:] - INFO [main:FourLetterWordMain@62] - connecting to 127.0.0.1 11289
2016-07-21 08:08:03,298 [myid:] - INFO [main:JMXEnv@146] - ensureOnly:[]
2016-07-21 08:08:03,384 [myid:] - INFO [main:ClientBase@443] - STARTING server
2016-07-21 08:08:03,385 [myid:] - INFO [main:ClientBase@364] - CREATING server instance 127.0.0.1:11289
2016-07-21 08:08:03,392 [myid:] - INFO [main:ClientBase@339] - STARTING server instance 127.0.0.1:11289
2016-07-21 08:08:03,392 [myid:] - INFO [main:ZooKeeperServer@170] - Created server with tickTime 3000 minSessionTimeout 6000 maxSessionTimeout 60000 datadir /home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/test/tmp/test4028127123136724556.junit.dir/version-2 snapdir /home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/test/tmp/test4028127123136724556.junit.dir/version-2
2016-07-21 08:08:03,392 [myid:] - INFO [main:NettyServerCnxnFactory@365] - binding to port 0.0.0.0/0.0.0.0:11289
2016-07-21 08:08:03,395 [myid:] - INFO [main:FourLetterWordMain@62] - connecting to 127.0.0.1 11289
2016-07-21 08:08:03,396 [myid:] - INFO [New I/O worker #2740:NettyServerCnxn@632] - Processing stat command from /127.0.0.1:36118
2016-07-21 08:08:03,396 [myid:] - INFO [New I/O worker #2740:NettyServerCnxn$StatCommand@469] - Stat command output
2016-07-21 08:08:03,397 [myid:] - INFO [main:JMXEnv@229] - ensureParent:[InMemoryDataTree, StandaloneServer_port]
2016-07-21 08:08:03,398 [myid:] - INFO [main:JMXEnv@246] - expect:InMemoryDataTree
2016-07-21 08:08:03,398 [myid:] - INFO [main:JMXEnv@250] - found:InMemoryDataTree org.apache.ZooKeeperService:name0=StandaloneServer_port11289,name1=InMemoryDataTree
2016-07-21 08:08:03,399 [myid:] - INFO [main:JMXEnv@246] - expect:StandaloneServer_port
2016-07-21 08:08:03,399 [myid:] - INFO [main:JMXEnv@250] - found:StandaloneServer_port org.apache.ZooKeeperService:name0=StandaloneServer_port11289
2016-07-21 08:08:04,956 [myid:] - INFO [main-SendThread(127.0.0.1:11289):ClientCnxn$SendThread@1032] - Opening socket connection to server 127.0.0.1/127.0.0.1:11289. Will not attempt to authenticate using SASL (unknown error)
2016-07-21 08:08:04,956 [myid:] - INFO [main-SendThread(127.0.0.1:11289):ClientCnxn$SendThread@876] - Socket connection established to 127.0.0.1/127.0.0.1:11289, initiating session
2016-07-21 08:08:04,957 [myid:] - INFO [New I/O worker #2741:ZooKeeperServer@893] - Client attempting to renew session 0x1560c7f3a630000 at /127.0.0.1:36123
2016-07-21 08:08:04,958 [myid:] - INFO [New I/O worker #2741:ZooKeeperServer@645] - Established session 0x1560c7f3a630000 with negotiated timeout 6000 for client /127.0.0.1:36123
2016-07-21 08:08:04,958 [myid:] - INFO [main-SendThread(127.0.0.1:11289):ClientCnxn$SendThread@1299] - Session establishment complete on server 127.0.0.1/127.0.0.1:11289, sessionid = 0x1560c7f3a630000, negotiated timeout = 6000
2016-07-21 08:08:04,959 [myid:] - INFO [SyncThread:0:FileTxnLog@203] - Creating new log file: log.7
2016-07-21 08:08:05,963 [myid:] - INFO [ProcessThread(sid:0 cport:11289)::PrepRequestProcessor@487] - Processed session termination for sessionid: 0x1560c7f3a630000
2016-07-21 08:08:05,964 [myid:] - INFO [main:ZooKeeper@684] - Session: 0x1560c7f3a630000 closed
2016-07-21 08:08:05,964 [myid:] - INFO [main-EventThread:ClientCnxn$EventThread@519] - EventThread shut down for session: 0x1560c7f3a630000
2016-07-21 08:08:05,965 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@58] - Memory used 71640
2016-07-21 08:08:05,965 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@63] - Number of threads 65
2016-07-21 08:08:05,965 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@78] - FINISHED TEST METHOD testWatcherAutoResetWithLocal
2016-07-21 08:08:05,965 [myid:] - INFO [main:ClientBase@520] - tearDown starting
2016-07-21 08:08:05,965 [myid:] - INFO [main:ClientBase@490] - STOPPING server
2016-07-21 08:08:05,965 [myid:] - INFO [main:NettyServerCnxnFactory@342] - shutdown called 0.0.0.0/0.0.0.0:11289
2016-07-21 08:08:05,972 [myid:] - INFO [main:ZooKeeperServer@469] - shutting down
2016-07-21 08:08:05,975 [myid:] - INFO [main:SessionTrackerImpl@225] - Shutting down
2016-07-21 08:08:05,975 [myid:] - INFO [main:PrepRequestProcessor@765] - Shutting down
2016-07-21 08:08:05,975 [myid:] - INFO [main:SyncRequestProcessor@209] - Shutting down
2016-07-21 08:08:05,975 [myid:] - INFO [ProcessThread(sid:0 cport:11289)::PrepRequestProcessor@143] - PrepRequestProcessor exited loop!
2016-07-21 08:08:05,975 [myid:] - INFO [SyncThread:0:SyncRequestProcessor@187] - SyncRequestProcessor exited!
2016-07-21 08:08:05,976 [myid:] - INFO [main:FinalRequestProcessor@402] - shutdown of request processor complete
2016-07-21 08:08:05,977 [myid:] - INFO [main:FourLetterWordMain@62] - connecting to 127.0.0.1 11289
2016-07-21 08:08:05,978 [myid:] - INFO [main:JMXEnv@146] - ensureOnly:[]
2016-07-21 08:08:05,981 [myid:] - INFO [main:ClientBase@545] - fdcount after test is: 45 at start it was 41
2016-07-21 08:08:05,981 [myid:] - INFO [main:ClientBase@547] - sleeping for 20 secs
2016-07-21 08:08:05,981 [myid:] - INFO [main:ZKTestCase$1@60] - SUCCEEDED testWatcherAutoResetWithLocal
2016-07-21 08:08:05,982 [myid:] - INFO [main:ZKTestCase$1@55] - FINISHED testWatcherAutoResetWithLocal
2016-07-21 08:08:05,983 [myid:] - INFO [main:PortAssignment@32] - assigning port 11290
2016-07-21 08:08:05,983 [myid:] - INFO [main:ZKTestCase$1@50] - STARTING testWatcherAutoResetDisabledWithGlobal
2016-07-21 08:08:05,983 [myid:] - INFO [main:ClientBase@425] - Initial fdcount is: 45
2016-07-21 08:08:05,991 [myid:] - INFO [main:ClientBase@443] - STARTING server
2016-07-21 08:08:05,992 [myid:] - INFO [main:ClientBase@364] - CREATING server instance 127.0.0.1:11290
2016-07-21 08:08:06,000 [myid:] - INFO [SessionTracker:SessionTrackerImpl@162] - SessionTrackerImpl exited loop!
2016-07-21 08:08:06,000 [myid:] - INFO [SessionTracker:SessionTrackerImpl@162] - SessionTrackerImpl exited loop!
2016-07-21 08:08:06,002 [myid:] - INFO [main:ClientBase@339] - STARTING server instance 127.0.0.1:11290
2016-07-21 08:08:06,002 [myid:] - INFO [main:ZooKeeperServer@170] - Created server with tickTime 3000 minSessionTimeout 6000 maxSessionTimeout 60000 datadir /home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/test/tmp/test6527239161822360616.junit.dir/version-2 snapdir /home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/test/tmp/test6527239161822360616.junit.dir/version-2
2016-07-21 08:08:06,002 [myid:] - INFO [main:NettyServerCnxnFactory@365] - binding to port 0.0.0.0/0.0.0.0:11290
2016-07-21 08:08:06,004 [myid:] - INFO [main:FourLetterWordMain@62] - connecting to 127.0.0.1 11290
2016-07-21 08:08:06,005 [myid:] - INFO [New I/O worker #2773:NettyServerCnxn@632] - Processing stat command from /127.0.0.1:59460
2016-07-21 08:08:06,005 [myid:] - INFO [New I/O worker #2773:NettyServerCnxn$StatCommand@469] - Stat command output
2016-07-21 08:08:06,006 [myid:] - INFO [main:JMXEnv@229] - ensureParent:[InMemoryDataTree, StandaloneServer_port]
2016-07-21 08:08:06,007 [myid:] - INFO [main:JMXEnv@246] - expect:InMemoryDataTree
2016-07-21 08:08:06,007 [myid:] - INFO [main:JMXEnv@250] - found:InMemoryDataTree org.apache.ZooKeeperService:name0=StandaloneServer_port11290,name1=InMemoryDataTree
2016-07-21 08:08:06,007 [myid:] - INFO [main:JMXEnv@246] - expect:StandaloneServer_port
2016-07-21 08:08:06,007 [myid:] - INFO [main:JMXEnv@250] - found:StandaloneServer_port org.apache.ZooKeeperService:name0=StandaloneServer_port11290
2016-07-21 08:08:06,007 [myid:] - INFO [main:ClientBase@439] - Client test setup finished
2016-07-21 08:08:06,008 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@53] - RUNNING TEST METHOD testWatcherAutoResetDisabledWithGlobal
2016-07-21 08:08:06,008 [myid:] - INFO [main:ZooKeeper@438] - Initiating client connection, connectString=127.0.0.1:11290 sessionTimeout=5000 watcher=org.apache.zookeeper.test.WatcherTest$MyWatcher@63979eb4
2016-07-21 08:08:06,009 [myid:] - INFO [main-SendThread(127.0.0.1:11290):ClientCnxn$SendThread@1032] - Opening socket connection to server 127.0.0.1/127.0.0.1:11290. Will not attempt to authenticate using SASL (unknown error)
2016-07-21 08:08:06,009 [myid:] - INFO [main-SendThread(127.0.0.1:11290):ClientCnxn$SendThread@876] - Socket connection established to 127.0.0.1/127.0.0.1:11290, initiating session
2016-07-21 08:08:06,010 [myid:] - INFO [New I/O worker #2774:ZooKeeperServer@900] - Client attempting to establish new session at /127.0.0.1:59461
2016-07-21 08:08:06,010 [myid:] - INFO [SyncThread:0:FileTxnLog@203] - Creating new log file: log.1
2016-07-21 08:08:06,012 [myid:] - INFO [SyncThread:0:ZooKeeperServer@645] - Established session 0x1560c7f52730000 with negotiated timeout 6000 for client /127.0.0.1:59461
2016-07-21 08:08:06,012 [myid:] - INFO [main-SendThread(127.0.0.1:11290):ClientCnxn$SendThread@1299] - Session establishment complete on server 127.0.0.1/127.0.0.1:11290, sessionid = 0x1560c7f52730000, negotiated timeout = 6000
2016-07-21 08:08:06,013 [myid:] - INFO [main:JMXEnv@117] - expect:0x1560c7f52730000
2016-07-21 08:08:06,013 [myid:] - INFO [main:JMXEnv@120] - found:0x1560c7f52730000 org.apache.ZooKeeperService:name0=StandaloneServer_port11290,name1=Connections,name2=127.0.0.1,name3=0x1560c7f52730000
2016-07-21 08:08:06,018 [myid:] - INFO [main:ClientBase@490] - STOPPING server
2016-07-21 08:08:06,019 [myid:] - INFO [main:NettyServerCnxnFactory@342] - shutdown called 0.0.0.0/0.0.0.0:11290
2016-07-21 08:08:06,019 [myid:] - INFO [main-SendThread(127.0.0.1:11290):ClientCnxn$SendThread@1158] - Unable to read additional data from server sessionid 0x1560c7f52730000, likely server has closed socket, closing socket connection and attempting reconnect
2016-07-21 08:08:06,023 [myid:] - INFO [main:ZooKeeperServer@469] - shutting down
2016-07-21 08:08:06,023 [myid:] - INFO [main:SessionTrackerImpl@225] - Shutting down
2016-07-21 08:08:06,024 [myid:] - INFO [main:PrepRequestProcessor@765] - Shutting down
2016-07-21 08:08:06,024 [myid:] - INFO [main:SyncRequestProcessor@209] - Shutting down
2016-07-21 08:08:06,024 [myid:] - INFO [ProcessThread(sid:0 cport:11290)::PrepRequestProcessor@143] - PrepRequestProcessor exited loop!
2016-07-21 08:08:06,024 [myid:] - INFO [SyncThread:0:SyncRequestProcessor@187] - SyncRequestProcessor exited!
2016-07-21 08:08:06,026 [myid:] - INFO [main:FinalRequestProcessor@402] - shutdown of request processor complete
2016-07-21 08:08:06,027 [myid:] - INFO [main:FourLetterWordMain@62] - connecting to 127.0.0.1 11290
2016-07-21 08:08:06,027 [myid:] - INFO [main:JMXEnv@146] - ensureOnly:[]
2016-07-21 08:08:06,119 [myid:] - INFO [main:ClientBase@443] - STARTING server
2016-07-21 08:08:06,120 [myid:] - INFO [main:ClientBase@364] - CREATING server instance 127.0.0.1:11290
2016-07-21 08:08:06,126 [myid:] - INFO [main:ClientBase@339] - STARTING server instance 127.0.0.1:11290
2016-07-21 08:08:06,127 [myid:] - INFO [main:ZooKeeperServer@170] - Created server with tickTime 3000 minSessionTimeout 6000 maxSessionTimeout 60000 datadir /home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/test/tmp/test6527239161822360616.junit.dir/version-2 snapdir /home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/test/tmp/test6527239161822360616.junit.dir/version-2
2016-07-21 08:08:06,127 [myid:] - INFO [main:NettyServerCnxnFactory@365] - binding to port 0.0.0.0/0.0.0.0:11290
2016-07-21 08:08:06,129 [myid:] - INFO [main:FourLetterWordMain@62] - connecting to 127.0.0.1 11290
2016-07-21 08:08:06,130 [myid:] - INFO [New I/O worker #2806:NettyServerCnxn@632] - Processing stat command from /127.0.0.1:59463
2016-07-21 08:08:06,130 [myid:] - INFO [New I/O worker #2806:NettyServerCnxn$StatCommand@469] - Stat command output
2016-07-21 08:08:06,131 [myid:] - INFO [main:JMXEnv@229] - ensureParent:[InMemoryDataTree, StandaloneServer_port]
2016-07-21 08:08:06,132 [myid:] - INFO [main:JMXEnv@246] - expect:InMemoryDataTree
2016-07-21 08:08:06,132 [myid:] - INFO [main:JMXEnv@250] - found:InMemoryDataTree org.apache.ZooKeeperService:name0=StandaloneServer_port11290,name1=InMemoryDataTree
2016-07-21 08:08:06,132 [myid:] - INFO [main:JMXEnv@246] - expect:StandaloneServer_port
2016-07-21 08:08:06,132 [myid:] - INFO [main:JMXEnv@250] - found:StandaloneServer_port org.apache.ZooKeeperService:name0=StandaloneServer_port11290
2016-07-21 08:08:07,861 [myid:] - INFO [main-SendThread(127.0.0.1:11290):ClientCnxn$SendThread@1032] - Opening socket connection to server 127.0.0.1/127.0.0.1:11290. Will not attempt to authenticate using SASL (unknown error)
2016-07-21 08:08:07,861 [myid:] - INFO [main-SendThread(127.0.0.1:11290):ClientCnxn$SendThread@876] - Socket connection established to 127.0.0.1/127.0.0.1:11290, initiating session
2016-07-21 08:08:07,862 [myid:] - INFO [New I/O worker #2807:ZooKeeperServer@893] - Client attempting to renew session 0x1560c7f52730000 at /127.0.0.1:59469
2016-07-21 08:08:07,862 [myid:] - INFO [New I/O worker #2807:ZooKeeperServer@645] - Established session 0x1560c7f52730000 with negotiated timeout 6000 for client /127.0.0.1:59469
2016-07-21 08:08:07,862 [myid:] - INFO [main-SendThread(127.0.0.1:11290):ClientCnxn$SendThread@1299] - Session establishment complete on server 127.0.0.1/127.0.0.1:11290, sessionid = 0x1560c7f52730000, negotiated timeout = 6000
2016-07-21 08:08:07,864 [myid:] - INFO [SyncThread:0:FileTxnLog@203] - Creating new log file: log.4
2016-07-21 08:08:07,868 [myid:] - INFO [main:ClientBase@490] - STOPPING server
2016-07-21 08:08:07,868 [myid:] - INFO [main:NettyServerCnxnFactory@342] - shutdown called 0.0.0.0/0.0.0.0:11290
2016-07-21 08:08:07,869 [myid:] - INFO [main-SendThread(127.0.0.1:11290):ClientCnxn$SendThread@1158] - Unable to read additional data from server sessionid 0x1560c7f52730000, likely server has closed socket, closing socket connection and attempting reconnect
2016-07-21 08:08:07,872 [myid:] - INFO [main:ZooKeeperServer@469] - shutting down
2016-07-21 08:08:07,872 [myid:] - INFO [main:SessionTrackerImpl@225] - Shutting down
2016-07-21 08:08:07,872 [myid:] - INFO [main:PrepRequestProcessor@765] - Shutting down
2016-07-21 08:08:07,873 [myid:] - INFO [main:SyncRequestProcessor@209] - Shutting down
2016-07-21 08:08:07,873 [myid:] - INFO [ProcessThread(sid:0 cport:11290)::PrepRequestProcessor@143] - PrepRequestProcessor exited loop!
2016-07-21 08:08:07,873 [myid:] - INFO [SyncThread:0:SyncRequestProcessor@187] - SyncRequestProcessor exited!
2016-07-21 08:08:07,874 [myid:] - INFO [main:FinalRequestProcessor@402] - shutdown of request processor complete
2016-07-21 08:08:07,874 [myid:] - INFO [main:FourLetterWordMain@62] - connecting to 127.0.0.1 11290
2016-07-21 08:08:07,875 [myid:] - INFO [main:JMXEnv@146] - ensureOnly:[]
2016-07-21 08:08:07,969 [myid:] - INFO [main:ClientBase@443] - STARTING server
2016-07-21 08:08:07,970 [myid:] - INFO [main:ClientBase@364] - CREATING server instance 127.0.0.1:11290
2016-07-21 08:08:07,978 [myid:] - INFO [main:ClientBase@339] - STARTING server instance 127.0.0.1:11290
2016-07-21 08:08:07,978 [myid:] - INFO [main:ZooKeeperServer@170] - Created server with tickTime 3000 minSessionTimeout 6000 maxSessionTimeout 60000 datadir /home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/test/tmp/test6527239161822360616.junit.dir/version-2 snapdir /home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/test/tmp/test6527239161822360616.junit.dir/version-2
2016-07-21 08:08:07,978 [myid:] - INFO [main:NettyServerCnxnFactory@365] - binding to port 0.0.0.0/0.0.0.0:11290
2016-07-21 08:08:07,981 [myid:] - INFO [main:FourLetterWordMain@62] - connecting to 127.0.0.1 11290
2016-07-21 08:08:07,982 [myid:] - INFO [New I/O worker #2839:NettyServerCnxn@632] - Processing stat command from /127.0.0.1:59471
2016-07-21 08:08:07,982 [myid:] - INFO [New I/O worker #2839:NettyServerCnxn$StatCommand@469] - Stat command output
2016-07-21 08:08:07,983 [myid:] - INFO [main:JMXEnv@229] - ensureParent:[InMemoryDataTree, StandaloneServer_port]
2016-07-21 08:08:07,984 [myid:] - INFO [main:JMXEnv@246] - expect:InMemoryDataTree
2016-07-21 08:08:07,985 [myid:] - INFO [main:JMXEnv@250] - found:InMemoryDataTree org.apache.ZooKeeperService:name0=StandaloneServer_port11290,name1=InMemoryDataTree
2016-07-21 08:08:07,985 [myid:] - INFO [main:JMXEnv@246] - expect:StandaloneServer_port
2016-07-21 08:08:07,985 [myid:] - INFO [main:JMXEnv@250] - found:StandaloneServer_port org.apache.ZooKeeperService:name0=StandaloneServer_port11290
2016-07-21 08:08:09,000 [myid:] - INFO [SessionTracker:SessionTrackerImpl@162] - SessionTrackerImpl exited loop!
2016-07-21 08:08:09,000 [myid:] - INFO [SessionTracker:SessionTrackerImpl@162] - SessionTrackerImpl exited loop!
2016-07-21 08:08:09,515 [myid:] - INFO [main-SendThread(127.0.0.1:11290):ClientCnxn$SendThread@1032] - Opening socket connection to server 127.0.0.1/127.0.0.1:11290. Will not attempt to authenticate using SASL (unknown error)
2016-07-21 08:08:09,515 [myid:] - INFO [main-SendThread(127.0.0.1:11290):ClientCnxn$SendThread@876] - Socket connection established to 127.0.0.1/127.0.0.1:11290, initiating session
2016-07-21 08:08:09,516 [myid:] - INFO [New I/O worker #2840:ZooKeeperServer@893] - Client attempting to renew session 0x1560c7f52730000 at /127.0.0.1:59477
2016-07-21 08:08:09,517 [myid:] - INFO [New I/O worker #2840:ZooKeeperServer@645] - Established session 0x1560c7f52730000 with negotiated timeout 6000 for client /127.0.0.1:59477
2016-07-21 08:08:09,517 [myid:] - INFO [main-SendThread(127.0.0.1:11290):ClientCnxn$SendThread@1299] - Session establishment complete on server 127.0.0.1/127.0.0.1:11290, sessionid = 0x1560c7f52730000, negotiated timeout = 6000
2016-07-21 08:08:09,521 [myid:] - INFO [SyncThread:0:FileTxnLog@203] - Creating new log file: log.6
2016-07-21 08:08:09,523 [myid:] - INFO [main:ClientBase@490] - STOPPING server
2016-07-21 08:08:09,523 [myid:] - INFO [main:NettyServerCnxnFactory@342] - shutdown called 0.0.0.0/0.0.0.0:11290
2016-07-21 08:08:09,524 [myid:] - INFO [main-SendThread(127.0.0.1:11290):ClientCnxn$SendThread@1158] - Unable to read additional data from server sessionid 0x1560c7f52730000, likely server has closed socket, closing socket connection and attempting reconnect
2016-07-21 08:08:09,530 [myid:] - INFO [main:ZooKeeperServer@469] - shutting down
2016-07-21 08:08:09,530 [myid:] - INFO [main:SessionTrackerImpl@225] - Shutting down
2016-07-21 08:08:09,530 [myid:] - INFO [main:PrepRequestProcessor@765] - Shutting down
2016-07-21 08:08:09,530 [myid:] - INFO [main:SyncRequestProcessor@209] - Shutting down
2016-07-21 08:08:09,531 [myid:] - INFO [ProcessThread(sid:0 cport:11290)::PrepRequestProcessor@143] - PrepRequestProcessor exited loop!
2016-07-21 08:08:09,531 [myid:] - INFO [SyncThread:0:SyncRequestProcessor@187] - SyncRequestProcessor exited!
2016-07-21 08:08:09,532 [myid:] - INFO [main:FinalRequestProcessor@402] - shutdown of request processor complete
2016-07-21 08:08:09,533 [myid:] - INFO [main:FourLetterWordMain@62] - connecting to 127.0.0.1 11290
2016-07-21 08:08:09,534 [myid:] - INFO [main:JMXEnv@146] - ensureOnly:[]
2016-07-21 08:08:09,625 [myid:] - INFO [main:ClientBase@443] - STARTING server
2016-07-21 08:08:09,625 [myid:] - INFO [main:ClientBase@364] - CREATING server instance 127.0.0.1:11290
2016-07-21 08:08:09,631 [myid:] - INFO [main:ClientBase@339] - STARTING server instance 127.0.0.1:11290
2016-07-21 08:08:09,632 [myid:] - INFO [main:ZooKeeperServer@170] - Created server with tickTime 3000 minSessionTimeout 6000 maxSessionTimeout 60000 datadir /home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/test/tmp/test6527239161822360616.junit.dir/version-2 snapdir /home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/test/tmp/test6527239161822360616.junit.dir/version-2
2016-07-21 08:08:09,632 [myid:] - INFO [main:NettyServerCnxnFactory@365] - binding to port 0.0.0.0/0.0.0.0:11290
2016-07-21 08:08:09,635 [myid:] - INFO [main:FourLetterWordMain@62] - connecting to 127.0.0.1 11290
2016-07-21 08:08:09,636 [myid:] - INFO [New I/O worker #2872:NettyServerCnxn@632] - Processing stat command from /127.0.0.1:59479
2016-07-21 08:08:09,636 [myid:] - INFO [New I/O worker #2872:NettyServerCnxn$StatCommand@469] - Stat command output
2016-07-21 08:08:09,637 [myid:] - INFO [main:JMXEnv@229] - ensureParent:[InMemoryDataTree, StandaloneServer_port]
2016-07-21 08:08:09,638 [myid:] - INFO [main:JMXEnv@246] - expect:InMemoryDataTree
2016-07-21 08:08:09,638 [myid:] - INFO [main:JMXEnv@250] - found:InMemoryDataTree org.apache.ZooKeeperService:name0=StandaloneServer_port11290,name1=InMemoryDataTree
2016-07-21 08:08:09,638 [myid:] - INFO [main:JMXEnv@246] - expect:StandaloneServer_port
2016-07-21 08:08:09,638 [myid:] - INFO [main:JMXEnv@250] - found:StandaloneServer_port org.apache.ZooKeeperService:name0=StandaloneServer_port11290
2016-07-21 08:08:11,158 [myid:] - INFO [main-SendThread(127.0.0.1:11290):ClientCnxn$SendThread@1032] - Opening socket connection to server 127.0.0.1/127.0.0.1:11290. Will not attempt to authenticate using SASL (unknown error)
2016-07-21 08:08:11,159 [myid:] - INFO [main-SendThread(127.0.0.1:11290):ClientCnxn$SendThread@876] - Socket connection established to 127.0.0.1/127.0.0.1:11290, initiating session
2016-07-21 08:08:11,160 [myid:] - INFO [New I/O worker #2873:ZooKeeperServer@893] - Client attempting to renew session 0x1560c7f52730000 at /127.0.0.1:59485
2016-07-21 08:08:11,161 [myid:] - INFO [New I/O worker #2873:ZooKeeperServer@645] - Established session 0x1560c7f52730000 with negotiated timeout 6000 for client /127.0.0.1:59485
2016-07-21 08:08:11,162 [myid:] - INFO [main-SendThread(127.0.0.1:11290):ClientCnxn$SendThread@1299] - Session establishment complete on server 127.0.0.1/127.0.0.1:11290, sessionid = 0x1560c7f52730000, negotiated timeout = 6000
2016-07-21 08:08:11,163 [myid:] - INFO [SyncThread:0:FileTxnLog@203] - Creating new log file: log.7
2016-07-21 08:08:12,000 [myid:] - INFO [SessionTracker:SessionTrackerImpl@162] - SessionTrackerImpl exited loop!
2016-07-21 08:08:12,166 [myid:] - INFO [ProcessThread(sid:0 cport:11290)::PrepRequestProcessor@487] - Processed session termination for sessionid: 0x1560c7f52730000
2016-07-21 08:08:12,167 [myid:] - INFO [main:ZooKeeper@684] - Session: 0x1560c7f52730000 closed
2016-07-21 08:08:12,167 [myid:] - INFO [main-EventThread:ClientCnxn$EventThread@519] - EventThread shut down for session: 0x1560c7f52730000
2016-07-21 08:08:12,168 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@58] - Memory used 93132
2016-07-21 08:08:12,168 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@63] - Number of threads 64
2016-07-21 08:08:12,168 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@78] - FINISHED TEST METHOD testWatcherAutoResetDisabledWithGlobal
2016-07-21 08:08:12,168 [myid:] - INFO [main:ClientBase@520] - tearDown starting
2016-07-21 08:08:12,168 [myid:] - INFO [main:ClientBase@490] - STOPPING server
2016-07-21 08:08:12,169 [myid:] - INFO [main:NettyServerCnxnFactory@342] - shutdown called 0.0.0.0/0.0.0.0:11290
2016-07-21 08:08:12,175 [myid:] - INFO [main:ZooKeeperServer@469] - shutting down
2016-07-21 08:08:12,176 [myid:] - INFO [main:SessionTrackerImpl@225] - Shutting down
2016-07-21 08:08:12,177 [myid:] - INFO [main:PrepRequestProcessor@765] - Shutting down
2016-07-21 08:08:12,177 [myid:] - INFO [main:SyncRequestProcessor@209] - Shutting down
2016-07-21 08:08:12,177 [myid:] - INFO [ProcessThread(sid:0 cport:11290)::PrepRequestProcessor@143] - PrepRequestProcessor exited loop!
2016-07-21 08:08:12,177 [myid:] - INFO [SyncThread:0:SyncRequestProcessor@187] - SyncRequestProcessor exited!
2016-07-21 08:08:12,178 [myid:] - INFO [main:FinalRequestProcessor@402] - shutdown of request processor complete
2016-07-21 08:08:12,179 [myid:] - INFO [main:FourLetterWordMain@62] - connecting to 127.0.0.1 11290
2016-07-21 08:08:12,179 [myid:] - INFO [main:JMXEnv@146] - ensureOnly:[]
2016-07-21 08:08:12,181 [myid:] - INFO [main:ClientBase@545] - fdcount after test is: 45 at start it was 45
2016-07-21 08:08:12,182 [myid:] - INFO [main:ZKTestCase$1@60] - SUCCEEDED testWatcherAutoResetDisabledWithGlobal
2016-07-21 08:08:12,182 [myid:] - INFO [main:ZKTestCase$1@55] - FINISHED testWatcherAutoResetDisabledWithGlobal
2016-07-21 08:08:12,182 [myid:] - INFO [main:PortAssignment@32] - assigning port 11291
2016-07-21 08:08:12,182 [myid:] - INFO [main:ZKTestCase$1@50] - STARTING testWatcherAutoResetDisabledWithLocal
2016-07-21 08:08:12,183 [myid:] - INFO [main:ClientBase@425] - Initial fdcount is: 45
2016-07-21 08:08:12,191 [myid:] - INFO [main:ClientBase@443] - STARTING server
2016-07-21 08:08:12,191 [myid:] - INFO [main:ClientBase@364] - CREATING server instance 127.0.0.1:11291
2016-07-21 08:08:12,197 [myid:] - INFO [main:ClientBase@339] - STARTING server instance 127.0.0.1:11291
2016-07-21 08:08:12,197 [myid:] - INFO [main:ZooKeeperServer@170] - Created server with tickTime 3000 minSessionTimeout 6000 maxSessionTimeout 60000 datadir /home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/test/tmp/test5697721892495919568.junit.dir/version-2 snapdir /home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/test/tmp/test5697721892495919568.junit.dir/version-2
2016-07-21 08:08:12,198 [myid:] - INFO [main:NettyServerCnxnFactory@365] - binding to port 0.0.0.0/0.0.0.0:11291
2016-07-21 08:08:12,199 [myid:] - INFO [main:FourLetterWordMain@62] - connecting to 127.0.0.1 11291
2016-07-21 08:08:12,200 [myid:] - INFO [New I/O worker #2905:NettyServerCnxn@632] - Processing stat command from /127.0.0.1:51830
2016-07-21 08:08:12,200 [myid:] - INFO [New I/O worker #2905:NettyServerCnxn$StatCommand@469] - Stat command output
2016-07-21 08:08:12,200 [myid:] - INFO [main:JMXEnv@229] - ensureParent:[InMemoryDataTree, StandaloneServer_port]
2016-07-21 08:08:12,202 [myid:] - INFO [main:JMXEnv@246] - expect:InMemoryDataTree
2016-07-21 08:08:12,202 [myid:] - INFO [main:JMXEnv@250] - found:InMemoryDataTree org.apache.ZooKeeperService:name0=StandaloneServer_port11291,name1=InMemoryDataTree
2016-07-21 08:08:12,202 [myid:] - INFO [main:JMXEnv@246] - expect:StandaloneServer_port
2016-07-21 08:08:12,202 [myid:] - INFO [main:JMXEnv@250] - found:StandaloneServer_port org.apache.ZooKeeperService:name0=StandaloneServer_port11291
2016-07-21 08:08:12,202 [myid:] - INFO [main:ClientBase@439] - Client test setup finished
2016-07-21 08:08:12,202 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@53] - RUNNING TEST METHOD testWatcherAutoResetDisabledWithLocal
2016-07-21 08:08:12,202 [myid:] - INFO [main:ZooKeeper@438] - Initiating client connection, connectString=127.0.0.1:11291 sessionTimeout=5000 watcher=org.apache.zookeeper.test.WatcherTest$MyWatcher@b901239
2016-07-21 08:08:12,203 [myid:] - INFO [main-SendThread(127.0.0.1:11291):ClientCnxn$SendThread@1032] - Opening socket connection to server 127.0.0.1/127.0.0.1:11291. Will not attempt to authenticate using SASL (unknown error)
2016-07-21 08:08:12,203 [myid:] - INFO [main-SendThread(127.0.0.1:11291):ClientCnxn$SendThread@876] - Socket connection established to 127.0.0.1/127.0.0.1:11291, initiating session
2016-07-21 08:08:12,204 [myid:] - INFO [New I/O worker #2906:ZooKeeperServer@900] - Client attempting to establish new session at /127.0.0.1:51831
2016-07-21 08:08:12,205 [myid:] - INFO [SyncThread:0:FileTxnLog@203] - Creating new log file: log.1
2016-07-21 08:08:12,206 [myid:] - INFO [SyncThread:0:ZooKeeperServer@645] - Established session 0x1560c7f6aa60000 with negotiated timeout 6000 for client /127.0.0.1:51831
2016-07-21 08:08:12,207 [myid:] - INFO [main-SendThread(127.0.0.1:11291):ClientCnxn$SendThread@1299] - Session establishment complete on server 127.0.0.1/127.0.0.1:11291, sessionid = 0x1560c7f6aa60000, negotiated timeout = 6000
2016-07-21 08:08:12,208 [myid:] - INFO [main:JMXEnv@117] - expect:0x1560c7f6aa60000
2016-07-21 08:08:12,209 [myid:] - INFO [main:JMXEnv@120] - found:0x1560c7f6aa60000 org.apache.ZooKeeperService:name0=StandaloneServer_port11291,name1=Connections,name2=127.0.0.1,name3=0x1560c7f6aa60000
2016-07-21 08:08:12,214 [myid:] - INFO [main:ClientBase@490] - STOPPING server
2016-07-21 08:08:12,214 [myid:] - INFO [main:NettyServerCnxnFactory@342] - shutdown called 0.0.0.0/0.0.0.0:11291
2016-07-21 08:08:12,214 [myid:] - INFO [main-SendThread(127.0.0.1:11291):ClientCnxn$SendThread@1158] - Unable to read additional data from server sessionid 0x1560c7f6aa60000, likely server has closed socket, closing socket connection and attempting reconnect
2016-07-21 08:08:12,219 [myid:] - INFO [main:ZooKeeperServer@469] - shutting down
2016-07-21 08:08:12,219 [myid:] - INFO [main:SessionTrackerImpl@225] - Shutting down
2016-07-21 08:08:12,219 [myid:] - INFO [main:PrepRequestProcessor@765] - Shutting down
2016-07-21 08:08:12,220 [myid:] - INFO [main:SyncRequestProcessor@209] - Shutting down
2016-07-21 08:08:12,220 [myid:] - INFO [ProcessThread(sid:0 cport:11291)::PrepRequestProcessor@143] - PrepRequestProcessor exited loop!
2016-07-21 08:08:12,220 [myid:] - INFO [SyncThread:0:SyncRequestProcessor@187] - SyncRequestProcessor exited!
2016-07-21 08:08:12,221 [myid:] - INFO [main:FinalRequestProcessor@402] - shutdown of request processor complete
2016-07-21 08:08:12,222 [myid:] - INFO [main:FourLetterWordMain@62] - connecting to 127.0.0.1 11291
2016-07-21 08:08:12,223 [myid:] - INFO [main:JMXEnv@146] - ensureOnly:[]
2016-07-21 08:08:12,315 [myid:] - INFO [main:ClientBase@443] - STARTING server
2016-07-21 08:08:12,315 [myid:] - INFO [main:ClientBase@364] - CREATING server instance 127.0.0.1:11291
2016-07-21 08:08:12,321 [myid:] - INFO [main:ClientBase@339] - STARTING server instance 127.0.0.1:11291
2016-07-21 08:08:12,321 [myid:] - INFO [main:ZooKeeperServer@170] - Created server with tickTime 3000 minSessionTimeout 6000 maxSessionTimeout 60000 datadir /home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/test/tmp/test5697721892495919568.junit.dir/version-2 snapdir /home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/test/tmp/test5697721892495919568.junit.dir/version-2
2016-07-21 08:08:12,321 [myid:] - INFO [main:NettyServerCnxnFactory@365] - binding to port 0.0.0.0/0.0.0.0:11291
2016-07-21 08:08:12,324 [myid:] - INFO [main:FourLetterWordMain@62] - connecting to 127.0.0.1 11291
2016-07-21 08:08:12,325 [myid:] - INFO [New I/O worker #2938:NettyServerCnxn@632] - Processing stat command from /127.0.0.1:51833
2016-07-21 08:08:12,325 [myid:] - INFO [New I/O worker #2938:NettyServerCnxn$StatCommand@469] - Stat command output
2016-07-21 08:08:12,326 [myid:] - INFO [main:JMXEnv@229] - ensureParent:[InMemoryDataTree, StandaloneServer_port]
2016-07-21 08:08:12,327 [myid:] - INFO [main:JMXEnv@246] - expect:InMemoryDataTree
2016-07-21 08:08:12,327 [myid:] - INFO [main:JMXEnv@250] - found:InMemoryDataTree org.apache.ZooKeeperService:name0=StandaloneServer_port11291,name1=InMemoryDataTree
2016-07-21 08:08:12,327 [myid:] - INFO [main:JMXEnv@246] - expect:StandaloneServer_port
2016-07-21 08:08:12,327 [myid:] - INFO [main:JMXEnv@250] - found:StandaloneServer_port org.apache.ZooKeeperService:name0=StandaloneServer_port11291
2016-07-21 08:08:13,518 [myid:] - INFO [main-SendThread(127.0.0.1:11291):ClientCnxn$SendThread@1032] - Opening socket connection to server 127.0.0.1/127.0.0.1:11291. Will not attempt to authenticate using SASL (unknown error)
2016-07-21 08:08:13,518 [myid:] - INFO [main-SendThread(127.0.0.1:11291):ClientCnxn$SendThread@876] - Socket connection established to 127.0.0.1/127.0.0.1:11291, initiating session
2016-07-21 08:08:13,519 [myid:] - INFO [New I/O worker #2939:ZooKeeperServer@893] - Client attempting to renew session 0x1560c7f6aa60000 at /127.0.0.1:51839
2016-07-21 08:08:13,520 [myid:] - INFO [New I/O worker #2939:ZooKeeperServer@645] - Established session 0x1560c7f6aa60000 with negotiated timeout 6000 for client /127.0.0.1:51839
2016-07-21 08:08:13,520 [myid:] - INFO [main-SendThread(127.0.0.1:11291):ClientCnxn$SendThread@1299] - Session establishment complete on server 127.0.0.1/127.0.0.1:11291, sessionid = 0x1560c7f6aa60000, negotiated timeout = 6000
2016-07-21 08:08:13,521 [myid:] - INFO [SyncThread:0:FileTxnLog@203] - Creating new log file: log.4
2016-07-21 08:08:13,523 [myid:] - INFO [main:ClientBase@490] - STOPPING server
2016-07-21 08:08:13,524 [myid:] - INFO [main:NettyServerCnxnFactory@342] - shutdown called 0.0.0.0/0.0.0.0:11291
2016-07-21 08:08:13,525 [myid:] - INFO [main-SendThread(127.0.0.1:11291):ClientCnxn$SendThread@1158] - Unable to read additional data from server sessionid 0x1560c7f6aa60000, likely server has closed socket, closing socket connection and attempting reconnect
2016-07-21 08:08:13,531 [myid:] - INFO [main:ZooKeeperServer@469] - shutting down
2016-07-21 08:08:13,531 [myid:] - INFO [main:SessionTrackerImpl@225] - Shutting down
2016-07-21 08:08:13,531 [myid:] - INFO [main:PrepRequestProcessor@765] - Shutting down
2016-07-21 08:08:13,531 [myid:] - INFO [main:SyncRequestProcessor@209] - Shutting down
2016-07-21 08:08:13,531 [myid:] - INFO [ProcessThread(sid:0 cport:11291)::PrepRequestProcessor@143] - PrepRequestProcessor exited loop!
2016-07-21 08:08:13,531 [myid:] - INFO [SyncThread:0:SyncRequestProcessor@187] - SyncRequestProcessor exited!
2016-07-21 08:08:13,532 [myid:] - INFO [main:FinalRequestProcessor@402] - shutdown of request processor complete
2016-07-21 08:08:13,533 [myid:] - INFO [main:FourLetterWordMain@62] - connecting to 127.0.0.1 11291
2016-07-21 08:08:13,533 [myid:] - INFO [main:JMXEnv@146] - ensureOnly:[]
2016-07-21 08:08:13,625 [myid:] - INFO [main:ClientBase@443] - STARTING server
2016-07-21 08:08:13,625 [myid:] - INFO [main:ClientBase@364] - CREATING server instance 127.0.0.1:11291
2016-07-21 08:08:13,633 [myid:] - INFO [main:ClientBase@339] - STARTING server instance 127.0.0.1:11291
2016-07-21 08:08:13,634 [myid:] - INFO [main:ZooKeeperServer@170] - Created server with tickTime 3000 minSessionTimeout 6000 maxSessionTimeout 60000 datadir /home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/test/tmp/test5697721892495919568.junit.dir/version-2 snapdir /home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/test/tmp/test5697721892495919568.junit.dir/version-2
2016-07-21 08:08:13,634 [myid:] - INFO [main:NettyServerCnxnFactory@365] - binding to port 0.0.0.0/0.0.0.0:11291
2016-07-21 08:08:13,636 [myid:] - INFO [main:FourLetterWordMain@62] - connecting to 127.0.0.1 11291
2016-07-21 08:08:13,637 [myid:] - INFO [New I/O worker #2971:NettyServerCnxn@632] - Processing stat command from /127.0.0.1:51841
2016-07-21 08:08:13,637 [myid:] - INFO [New I/O worker #2971:NettyServerCnxn$StatCommand@469] - Stat command output
2016-07-21 08:08:13,637 [myid:] - INFO [main:JMXEnv@229] - ensureParent:[InMemoryDataTree, StandaloneServer_port]
2016-07-21 08:08:13,639 [myid:] - INFO [main:JMXEnv@246] - expect:InMemoryDataTree
2016-07-21 08:08:13,639 [myid:] - INFO [main:JMXEnv@250] - found:InMemoryDataTree org.apache.ZooKeeperService:name0=StandaloneServer_port11291,name1=InMemoryDataTree
2016-07-21 08:08:13,639 [myid:] - INFO [main:JMXEnv@246] - expect:StandaloneServer_port
2016-07-21 08:08:13,639 [myid:] - INFO [main:JMXEnv@250] - found:StandaloneServer_port org.apache.ZooKeeperService:name0=StandaloneServer_port11291
2016-07-21 08:08:14,760 [myid:] - INFO [main-SendThread(127.0.0.1:11291):ClientCnxn$SendThread@1032] - Opening socket connection to server 127.0.0.1/127.0.0.1:11291. Will not attempt to authenticate using SASL (unknown error)
2016-07-21 08:08:14,761 [myid:] - INFO [main-SendThread(127.0.0.1:11291):ClientCnxn$SendThread@876] - Socket connection established to 127.0.0.1/127.0.0.1:11291, initiating session
2016-07-21 08:08:14,762 [myid:] - INFO [New I/O worker #2972:ZooKeeperServer@893] - Client attempting to renew session 0x1560c7f6aa60000 at /127.0.0.1:51846
2016-07-21 08:08:14,763 [myid:] - INFO [New I/O worker #2972:ZooKeeperServer@645] - Established session 0x1560c7f6aa60000 with negotiated timeout 6000 for client /127.0.0.1:51846
2016-07-21 08:08:14,763 [myid:] - INFO [main-SendThread(127.0.0.1:11291):ClientCnxn$SendThread@1299] - Session establishment complete on server 127.0.0.1/127.0.0.1:11291, sessionid = 0x1560c7f6aa60000, negotiated timeout = 6000
2016-07-21 08:08:14,768 [myid:] - INFO [SyncThread:0:FileTxnLog@203] - Creating new log file: log.6
2016-07-21 08:08:14,770 [myid:] - INFO [main:ClientBase@490] - STOPPING server
2016-07-21 08:08:14,771 [myid:] - INFO [main:NettyServerCnxnFactory@342] - shutdown called 0.0.0.0/0.0.0.0:11291
2016-07-21 08:08:14,771 [myid:] - INFO [main-SendThread(127.0.0.1:11291):ClientCnxn$SendThread@1158] - Unable to read additional data from server sessionid 0x1560c7f6aa60000, likely server has closed socket, closing socket connection and attempting reconnect
2016-07-21 08:08:14,777 [myid:] - INFO [main:ZooKeeperServer@469] - shutting down
2016-07-21 08:08:14,777 [myid:] - INFO [main:SessionTrackerImpl@225] - Shutting down
2016-07-21 08:08:14,778 [myid:] - INFO [main:PrepRequestProcessor@765] - Shutting down
2016-07-21 08:08:14,779 [myid:] - INFO [main:SyncRequestProcessor@209] - Shutting down
2016-07-21 08:08:14,779 [myid:] - INFO [ProcessThread(sid:0 cport:11291)::PrepRequestProcessor@143] - PrepRequestProcessor exited loop!
2016-07-21 08:08:14,779 [myid:] - INFO [SyncThread:0:SyncRequestProcessor@187] - SyncRequestProcessor exited!
2016-07-21 08:08:14,780 [myid:] - INFO [main:FinalRequestProcessor@402] - shutdown of request processor complete
2016-07-21 08:08:14,782 [myid:] - INFO [main:FourLetterWordMain@62] - connecting to 127.0.0.1 11291
2016-07-21 08:08:14,782 [myid:] - INFO [main:JMXEnv@146] - ensureOnly:[]
2016-07-21 08:08:14,871 [myid:] - INFO [main:ClientBase@443] - STARTING server
2016-07-21 08:08:14,872 [myid:] - INFO [main:ClientBase@364] - CREATING server instance 127.0.0.1:11291
2016-07-21 08:08:14,880 [myid:] - INFO [main:ClientBase@339] - STARTING server instance 127.0.0.1:11291
2016-07-21 08:08:14,881 [myid:] - INFO [main:ZooKeeperServer@170] - Created server with tickTime 3000 minSessionTimeout 6000 maxSessionTimeout 60000 datadir /home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/test/tmp/test5697721892495919568.junit.dir/version-2 snapdir /home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/test/tmp/test5697721892495919568.junit.dir/version-2
2016-07-21 08:08:14,881 [myid:] - INFO [main:NettyServerCnxnFactory@365] - binding to port 0.0.0.0/0.0.0.0:11291
2016-07-21 08:08:14,884 [myid:] - INFO [main:FourLetterWordMain@62] - connecting to 127.0.0.1 11291
2016-07-21 08:08:14,885 [myid:] - INFO [New I/O worker #3004:NettyServerCnxn@632] - Processing stat command from /127.0.0.1:51849
2016-07-21 08:08:14,886 [myid:] - INFO [New I/O worker #3004:NettyServerCnxn$StatCommand@469] - Stat command output
2016-07-21 08:08:14,886 [myid:] - INFO [main:JMXEnv@229] - ensureParent:[InMemoryDataTree, StandaloneServer_port]
2016-07-21 08:08:14,887 [myid:] - INFO [main:JMXEnv@246] - expect:InMemoryDataTree
2016-07-21 08:08:14,887 [myid:] - INFO [main:JMXEnv@250] - found:InMemoryDataTree org.apache.ZooKeeperService:name0=StandaloneServer_port11291,name1=InMemoryDataTree
2016-07-21 08:08:14,887 [myid:] - INFO [main:JMXEnv@246] - expect:StandaloneServer_port
2016-07-21 08:08:14,888 [myid:] - INFO [main:JMXEnv@250] - found:StandaloneServer_port org.apache.ZooKeeperService:name0=StandaloneServer_port11291
2016-07-21 08:08:15,000 [myid:] - INFO [SessionTracker:SessionTrackerImpl@162] - SessionTrackerImpl exited loop!
2016-07-21 08:08:15,001 [myid:] - INFO [SessionTracker:SessionTrackerImpl@162] - SessionTrackerImpl exited loop!
2016-07-21 08:08:15,001 [myid:] - INFO [SessionTracker:SessionTrackerImpl@162] - SessionTrackerImpl exited loop!
2016-07-21 08:08:15,001 [myid:] - INFO [SessionTracker:SessionTrackerImpl@162] - SessionTrackerImpl exited loop!
2016-07-21 08:08:16,075 [myid:] - INFO [main-SendThread(127.0.0.1:11291):ClientCnxn$SendThread@1032] - Opening socket connection to server 127.0.0.1/127.0.0.1:11291. Will not attempt to authenticate using SASL (unknown error)
2016-07-21 08:08:16,075 [myid:] - INFO [main-SendThread(127.0.0.1:11291):ClientCnxn$SendThread@876] - Socket connection established to 127.0.0.1/127.0.0.1:11291, initiating session
2016-07-21 08:08:16,076 [myid:] - INFO [New I/O worker #3005:ZooKeeperServer@893] - Client attempting to renew session 0x1560c7f6aa60000 at /127.0.0.1:51852
2016-07-21 08:08:16,077 [myid:] - INFO [New I/O worker #3005:ZooKeeperServer@645] - Established session 0x1560c7f6aa60000 with negotiated timeout 6000 for client /127.0.0.1:51852
2016-07-21 08:08:16,077 [myid:] - INFO [main-SendThread(127.0.0.1:11291):ClientCnxn$SendThread@1299] - Session establishment complete on server 127.0.0.1/127.0.0.1:11291, sessionid = 0x1560c7f6aa60000, negotiated timeout = 6000
2016-07-21 08:08:16,078 [myid:] - INFO [SyncThread:0:FileTxnLog@203] - Creating new log file: log.7
2016-07-21 08:08:17,082 [myid:] - INFO [ProcessThread(sid:0 cport:11291)::PrepRequestProcessor@487] - Processed session termination for sessionid: 0x1560c7f6aa60000
2016-07-21 08:08:17,083 [myid:] - WARN [New I/O worker #3005:NettyServerCnxnFactory$CnxnChannelHandler@111] - Exception caught [id: 0x3265227a, /127.0.0.1:51852 :> /127.0.0.1:11291] EXCEPTION: java.nio.channels.ClosedChannelException
java.nio.channels.ClosedChannelException
at sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:270)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:479)
at org.jboss.netty.channel.socket.nio.SocketSendBufferPool$UnpooledSendBuffer.transferTo(SocketSendBufferPool.java:203)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:201)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromTaskLoop(AbstractNioWorker.java:151)
at org.jboss.netty.channel.socket.nio.AbstractNioChannel$WriteTask.run(AbstractNioChannel.java:315)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:391)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:315)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2016-07-21 08:08:17,184 [myid:] - INFO [main:ZooKeeper@684] - Session: 0x1560c7f6aa60000 closed
2016-07-21 08:08:17,184 [myid:] - INFO [main-EventThread:ClientCnxn$EventThread@519] - EventThread shut down for session: 0x1560c7f6aa60000
2016-07-21 08:08:17,184 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@58] - Memory used 115316
2016-07-21 08:08:17,184 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@63] - Number of threads 63
2016-07-21 08:08:17,184 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@78] - FINISHED TEST METHOD testWatcherAutoResetDisabledWithLocal
2016-07-21 08:08:17,184 [myid:] - INFO [main:ClientBase@520] - tearDown starting
2016-07-21 08:08:17,184 [myid:] - INFO [main:ClientBase@490] - STOPPING server
2016-07-21 08:08:17,185 [myid:] - INFO [main:NettyServerCnxnFactory@342] - shutdown called 0.0.0.0/0.0.0.0:11291
2016-07-21 08:08:17,190 [myid:] - INFO [main:ZooKeeperServer@469] - shutting down
2016-07-21 08:08:17,191 [myid:] - INFO [main:SessionTrackerImpl@225] - Shutting down
2016-07-21 08:08:17,191 [myid:] - INFO [main:PrepRequestProcessor@765] - Shutting down
2016-07-21 08:08:17,191 [myid:] - INFO [main:SyncRequestProcessor@209] - Shutting down
2016-07-21 08:08:17,191 [myid:] - INFO [ProcessThread(sid:0 cport:11291)::PrepRequestProcessor@143] - PrepRequestProcessor exited loop!
2016-07-21 08:08:17,191 [myid:] - INFO [SyncThread:0:SyncRequestProcessor@187] - SyncRequestProcessor exited!
2016-07-21 08:08:17,192 [myid:] - INFO [main:FinalRequestProcessor@402] - shutdown of request processor complete
2016-07-21 08:08:17,192 [myid:] - INFO [main:FourLetterWordMain@62] - connecting to 127.0.0.1 11291
2016-07-21 08:08:17,193 [myid:] - INFO [main:JMXEnv@146] - ensureOnly:[]
2016-07-21 08:08:17,195 [myid:] - INFO [main:ClientBase@545] - fdcount after test is: 41 at start it was 45
2016-07-21 08:08:17,195 [myid:] - INFO [main:ZKTestCase$1@60] - SUCCEEDED testWatcherAutoResetDisabledWithLocal
2016-07-21 08:08:17,195 [myid:] - INFO [main:ZKTestCase$1@55] - FINISHED testWatcherAutoResetDisabledWithLocal
{noformat}

flaky, flaky-test 9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
1 year, 21 weeks ago 0|i31cm7:
ZooKeeper ZOOKEEPER-2484

Flaky Test: org.apache.zookeeper.test.LoadFromLogTest.testLoadFailure

Test Closed Major Fixed Michael Han Michael Han Michael Han 21/Jul/16 17:55   17/May/17 23:43 08/Sep/16 18:11 3.4.8, 3.5.2 3.5.3, 3.6.0 server, tests   0 2   From https://builds.apache.org/job/ZooKeeper-trunk-openjdk7/1098/

{noformat}
Error Message

KeeperErrorCode = ConnectionLoss for /data-
Stacktrace

org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /data-
at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:1412)
at org.apache.zookeeper.test.LoadFromLogTest.testLoadFailure(LoadFromLogTest.java:157)
at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:79)
Standard Output

2016-07-21 04:29:02,537 [myid:] - INFO [main:PortAssignment@157] - Single test process using ports from 11221 - 32767.
2016-07-21 04:29:02,551 [myid:] - INFO [main:PortAssignment@85] - Assigned port 11222 from range 11221 - 32767.
2016-07-21 04:29:02,652 [myid:] - INFO [main:JUnit4ZKTestRunner@47] - No test.method specified. using default methods.
2016-07-21 04:29:02,719 [myid:] - INFO [main:JUnit4ZKTestRunner@47] - No test.method specified. using default methods.
2016-07-21 04:29:02,733 [myid:] - INFO [main:ZKTestCase$1@55] - STARTING testReloadSnapshotWithMissingParent
2016-07-21 04:29:02,734 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@77] - RUNNING TEST METHOD testReloadSnapshotWithMissingParent
2016-07-21 04:29:02,768 [myid:] - INFO [main:Environment@109] - Server environment:zookeeper.version=3.6.0-SNAPSHOT-1753636, built on 07/21/2016 03:58 GMT
2016-07-21 04:29:02,768 [myid:] - INFO [main:Environment@109] - Server environment:host.name=jenkins-test-3b9
2016-07-21 04:29:02,768 [myid:] - INFO [main:Environment@109] - Server environment:java.version=1.7.0_101
2016-07-21 04:29:02,769 [myid:] - INFO [main:Environment@109] - Server environment:java.vendor=Oracle Corporation
2016-07-21 04:29:02,769 [myid:] - INFO [main:Environment@109] - Server environment:java.home=/usr/lib/jvm/java-7-openjdk-amd64/jre
2016-07-21 04:29:02,769 [myid:] - INFO [main:Environment@109] - Server environment:java.class.path=/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/trunk/build/test/classes:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/trunk/build/test/lib/antlr-2.7.7.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/trunk/build/test/lib/antlr4-runtime-4.5.1-1.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/trunk/build/test/lib/checkstyle-6.13.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/trunk/build/test/lib/commons-beanutils-1.9.2.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/trunk/build/test/lib/commons-cli-1.3.1.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/trunk/build/test/lib/commons-collections-3.2.2.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/trunk/build/test/lib/commons-lang3-3.4.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/trunk/build/test/lib/commons-logging-1.1.1.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/trunk/build/test/lib/guava-18.0.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/trunk/build/test/lib/hamcrest-core-1.3.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/trunk/build/test/lib/junit-4.12.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/trunk/build/test/lib/mockito-all-1.8.2.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/trunk/build/classes:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/trunk/src/java/lib/ivy-2.4.0.jar:/home/jenkins/tools/ant/latest/lib/ant.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/trunk/build/lib/commons-cli-1.2.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/trunk/build/lib/jackson-core-asl-1.9.11.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/trunk/build/lib/jackson-mapper-asl-1.9.11.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/trunk/build/lib/javacc.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/trunk/build/lib/jetty-6.1.26.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/trunk/build/lib/jetty-util-6.1.26.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/trunk/build/lib/jline-2.11.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/trunk/build/lib/log4j-1.2.17.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/trunk/build/lib/netty-3.10.5.Final.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/trunk/build/lib/servlet-api-2.5-20081211.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/trunk/build/lib/slf4j-api-1.7.5.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/trunk/build/lib/slf4j-log4j12-1.7.5.jar:/home/jenkins/tools/ant/apache-ant-1.8.2/lib/ant-launcher.jar:/home/jenkins/tools/ant/latest/lib/ant-junit.jar:/home/jenkins/tools/ant/latest/lib/ant-junit4.jar
2016-07-21 04:29:02,769 [myid:] - INFO [main:Environment@109] - Server environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib
2016-07-21 04:29:02,769 [myid:] - INFO [main:Environment@109] - Server environment:java.io.tmpdir=/tmp
2016-07-21 04:29:02,770 [myid:] - INFO [main:Environment@109] - Server environment:java.compiler=<NA>
2016-07-21 04:29:02,770 [myid:] - INFO [main:Environment@109] - Server environment:os.name=Linux
2016-07-21 04:29:02,772 [myid:] - INFO [main:Environment@109] - Server environment:os.arch=amd64
2016-07-21 04:29:02,773 [myid:] - INFO [main:Environment@109] - Server environment:os.version=3.13.0-30-generic
2016-07-21 04:29:02,773 [myid:] - INFO [main:Environment@109] - Server environment:user.name=jenkins
2016-07-21 04:29:02,773 [myid:] - INFO [main:Environment@109] - Server environment:user.home=/home/jenkins
2016-07-21 04:29:02,773 [myid:] - INFO [main:Environment@109] - Server environment:user.dir=/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/trunk/build/test
2016-07-21 04:29:02,773 [myid:] - INFO [main:Environment@109] - Server environment:os.memory.free=48MB
2016-07-21 04:29:02,773 [myid:] - INFO [main:Environment@109] - Server environment:os.memory.max=455MB
2016-07-21 04:29:02,774 [myid:] - INFO [main:Environment@109] - Server environment:os.memory.total=60MB
2016-07-21 04:29:02,790 [myid:] - INFO [main:ZooKeeperServer@858] - minSessionTimeout set to 6000
2016-07-21 04:29:02,790 [myid:] - INFO [main:ZooKeeperServer@867] - maxSessionTimeout set to 60000
2016-07-21 04:29:02,790 [myid:] - INFO [main:ZooKeeperServer@156] - Created server with tickTime 3000 minSessionTimeout 6000 maxSessionTimeout 60000 datadir /home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/trunk/build/test/tmp/test393292139818745013.junit.dir/version-2 snapdir /home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/trunk/build/test/tmp/test393292139818745013.junit.dir/version-2
2016-07-21 04:29:02,811 [myid:] - INFO [main:NIOServerCnxnFactory@673] - Configuring NIO connection handler with 10s sessionless connection timeout, 1 selector thread(s), 8 worker threads, and 64 kB direct buffers.
2016-07-21 04:29:02,821 [myid:] - INFO [main:NIOServerCnxnFactory@686] - binding to port 0.0.0.0/0.0.0.0:11222
2016-07-21 04:29:02,835 [myid:] - INFO [main:FileTxnSnapLog@298] - Snapshotting: 0x0 to /home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/trunk/build/test/tmp/test393292139818745013.junit.dir/version-2/snapshot.0
2016-07-21 04:29:02,952 [myid:] - INFO [main:FourLetterWordMain@85] - connecting to 127.0.0.1 11222
2016-07-21 04:29:02,958 [myid:] - INFO [NIOServerCxnFactory.AcceptThread:0.0.0.0/0.0.0.0:11222:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:48675
2016-07-21 04:29:02,967 [myid:] - INFO [NIOWorkerThread-1:NIOServerCnxn@485] - Processing stat command from /127.0.0.1:48675
2016-07-21 04:29:02,984 [myid:] - INFO [NIOWorkerThread-1:StatCommand@49] - Stat command output
2016-07-21 04:29:02,985 [myid:] - INFO [NIOWorkerThread-1:NIOServerCnxn@607] - Closed socket connection for client /127.0.0.1:48675 (no session established for client)
2016-07-21 04:29:02,992 [myid:] - INFO [main:Environment@109] - Client environment:zookeeper.version=3.6.0-SNAPSHOT-1753636, built on 07/21/2016 03:58 GMT
2016-07-21 04:29:02,992 [myid:] - INFO [main:Environment@109] - Client environment:host.name=jenkins-test-3b9
2016-07-21 04:29:02,992 [myid:] - INFO [main:Environment@109] - Client environment:java.version=1.7.0_101
2016-07-21 04:29:02,992 [myid:] - INFO [main:Environment@109] - Client environment:java.vendor=Oracle Corporation
2016-07-21 04:29:02,993 [myid:] - INFO [main:Environment@109] - Client environment:java.home=/usr/lib/jvm/java-7-openjdk-amd64/jre
2016-07-21 04:29:02,993 [myid:] - INFO [main:Environment@109] - Client environment:java.class.path=/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/trunk/build/test/classes:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/trunk/build/test/lib/antlr-2.7.7.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/trunk/build/test/lib/antlr4-runtime-4.5.1-1.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/trunk/build/test/lib/checkstyle-6.13.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/trunk/build/test/lib/commons-beanutils-1.9.2.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/trunk/build/test/lib/commons-cli-1.3.1.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/trunk/build/test/lib/commons-collections-3.2.2.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/trunk/build/test/lib/commons-lang3-3.4.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/trunk/build/test/lib/commons-logging-1.1.1.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/trunk/build/test/lib/guava-18.0.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/trunk/build/test/lib/hamcrest-core-1.3.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/trunk/build/test/lib/junit-4.12.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/trunk/build/test/lib/mockito-all-1.8.2.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/trunk/build/classes:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/trunk/src/java/lib/ivy-2.4.0.jar:/home/jenkins/tools/ant/latest/lib/ant.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/trunk/build/lib/commons-cli-1.2.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/trunk/build/lib/jackson-core-asl-1.9.11.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/trunk/build/lib/jackson-mapper-asl-1.9.11.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/trunk/build/lib/javacc.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/trunk/build/lib/jetty-6.1.26.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/trunk/build/lib/jetty-util-6.1.26.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/trunk/build/lib/jline-2.11.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/trunk/build/lib/log4j-1.2.17.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/trunk/build/lib/netty-3.10.5.Final.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/trunk/build/lib/servlet-api-2.5-20081211.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/trunk/build/lib/slf4j-api-1.7.5.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/trunk/build/lib/slf4j-log4j12-1.7.5.jar:/home/jenkins/tools/ant/apache-ant-1.8.2/lib/ant-launcher.jar:/home/jenkins/tools/ant/latest/lib/ant-junit.jar:/home/jenkins/tools/ant/latest/lib/ant-junit4.jar
2016-07-21 04:29:02,993 [myid:] - INFO [main:Environment@109] - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib
2016-07-21 04:29:02,993 [myid:] - INFO [main:Environment@109] - Client environment:java.io.tmpdir=/tmp
2016-07-21 04:29:02,994 [myid:] - INFO [main:Environment@109] - Client environment:java.compiler=<NA>
2016-07-21 04:29:02,994 [myid:] - INFO [main:Environment@109] - Client environment:os.name=Linux
2016-07-21 04:29:02,994 [myid:] - INFO [main:Environment@109] - Client environment:os.arch=amd64
2016-07-21 04:29:02,994 [myid:] - INFO [main:Environment@109] - Client environment:os.version=3.13.0-30-generic
2016-07-21 04:29:02,994 [myid:] - INFO [main:Environment@109] - Client environment:user.name=jenkins
2016-07-21 04:29:02,994 [myid:] - INFO [main:Environment@109] - Client environment:user.home=/home/jenkins
2016-07-21 04:29:02,995 [myid:] - INFO [main:Environment@109] - Client environment:user.dir=/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/trunk/build/test
2016-07-21 04:29:02,995 [myid:] - INFO [main:Environment@109] - Client environment:os.memory.free=56MB
2016-07-21 04:29:02,995 [myid:] - INFO [main:Environment@109] - Client environment:os.memory.max=455MB
2016-07-21 04:29:02,995 [myid:] - INFO [main:Environment@109] - Client environment:os.memory.total=60MB
2016-07-21 04:29:02,998 [myid:] - INFO [main:ZooKeeper@855] - Initiating client connection, connectString=127.0.0.1:11222 sessionTimeout=3000 watcher=org.apache.zookeeper.test.LoadFromLogTest@20772fd3
2016-07-21 04:29:03,020 [myid:127.0.0.1:11222] - INFO [main-SendThread(127.0.0.1:11222):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:11222. Will not attempt to authenticate using SASL (unknown error)
2016-07-21 04:29:03,021 [myid:] - INFO [NIOServerCxnFactory.AcceptThread:0.0.0.0/0.0.0.0:11222:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:48676
2016-07-21 04:29:03,021 [myid:127.0.0.1:11222] - INFO [main-SendThread(127.0.0.1:11222):ClientCnxn$SendThread@948] - Socket connection established, initiating session, client: /127.0.0.1:48676, server: 127.0.0.1/127.0.0.1:11222
2016-07-21 04:29:03,026 [myid:] - INFO [NIOWorkerThread-2:ZooKeeperServer@964] - Client attempting to establish new session at /127.0.0.1:48676
2016-07-21 04:29:03,030 [myid:] - INFO [SyncThread:0:FileTxnLog@204] - Creating new log file: log.1
2016-07-21 04:29:03,049 [myid:] - INFO [SyncThread:0:ZooKeeperServer@678] - Established session 0x10002e28feb0000 with negotiated timeout 6000 for client /127.0.0.1:48676
2016-07-21 04:29:03,051 [myid:127.0.0.1:11222] - INFO [main-SendThread(127.0.0.1:11222):ClientCnxn$SendThread@1381] - Session establishment complete on server 127.0.0.1/127.0.0.1:11222, sessionid = 0x10002e28feb0000, negotiated timeout = 6000
2016-07-21 04:29:03,244 [myid:] - INFO [main:LoadFromLogTest@553] - Set lastProcessedZxid to 2
2016-07-21 04:29:03,244 [myid:] - INFO [main:FileTxnSnapLog@298] - Snapshotting: 0x2 to /home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/trunk/build/test/tmp/test393292139818745013.junit.dir/version-2/snapshot.2
2016-07-21 04:29:03,245 [myid:] - INFO [main:ZooKeeperServer@498] - shutting down
2016-07-21 04:29:03,245 [myid:] - INFO [main:SessionTrackerImpl@232] - Shutting down
2016-07-21 04:29:03,245 [myid:] - INFO [main:PrepRequestProcessor@965] - Shutting down
2016-07-21 04:29:03,245 [myid:] - INFO [main:SyncRequestProcessor@191] - Shutting down
2016-07-21 04:29:03,245 [myid:] - INFO [ProcessThread(sid:0 cport:11222)::PrepRequestProcessor@154] - PrepRequestProcessor exited loop!
2016-07-21 04:29:03,245 [myid:] - INFO [SyncThread:0:SyncRequestProcessor@169] - SyncRequestProcessor exited!
2016-07-21 04:29:03,246 [myid:] - INFO [main:FinalRequestProcessor@479] - shutdown of request processor complete
2016-07-21 04:29:03,246 [myid:] - INFO [main:MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=StandaloneServer_port11222,name1=InMemoryDataTree]
2016-07-21 04:29:03,246 [myid:] - INFO [main:MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=StandaloneServer_port11222]
2016-07-21 04:29:03,247 [myid:] - INFO [ConnnectionExpirer:NIOServerCnxnFactory$ConnectionExpirerThread@583] - ConnnectionExpirerThread interrupted
2016-07-21 04:29:03,247 [myid:] - INFO [NIOServerCxnFactory.SelectorThread-0:MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=StandaloneServer_port11222,name1=Connections,name2=127.0.0.1,name3=0x10002e28feb0000]
2016-07-21 04:29:03,247 [myid:] - INFO [NIOServerCxnFactory.SelectorThread-0:NIOServerCnxn@607] - Closed socket connection for client /127.0.0.1:48676 which had sessionid 0x10002e28feb0000
2016-07-21 04:29:03,247 [myid:] - INFO [NIOServerCxnFactory.AcceptThread:0.0.0.0/0.0.0.0:11222:NIOServerCnxnFactory$AcceptThread@219] - accept thread exitted run method
2016-07-21 04:29:03,248 [myid:127.0.0.1:11222] - INFO [main-SendThread(127.0.0.1:11222):ClientCnxn$SendThread@1231] - Unable to read additional data from server sessionid 0x10002e28feb0000, likely server has closed socket, closing socket connection and attempting reconnect
2016-07-21 04:29:03,248 [myid:] - INFO [NIOServerCxnFactory.SelectorThread-0:NIOServerCnxnFactory$SelectorThread@420] - selector thread exitted run method
2016-07-21 04:29:03,248 [myid:] - INFO [main:ZooKeeperServer@858] - minSessionTimeout set to 6000
2016-07-21 04:29:03,249 [myid:] - INFO [main:ZooKeeperServer@867] - maxSessionTimeout set to 60000
2016-07-21 04:29:03,249 [myid:] - INFO [main:ZooKeeperServer@156] - Created server with tickTime 3000 minSessionTimeout 6000 maxSessionTimeout 60000 datadir /home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/trunk/build/test/tmp/test393292139818745013.junit.dir/version-2 snapdir /home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/trunk/build/test/tmp/test393292139818745013.junit.dir/version-2
2016-07-21 04:29:03,249 [myid:] - INFO [main:NIOServerCnxnFactory@673] - Configuring NIO connection handler with 10s sessionless connection timeout, 1 selector thread(s), 8 worker threads, and 64 kB direct buffers.
2016-07-21 04:29:03,249 [myid:] - INFO [main:NIOServerCnxnFactory@686] - binding to port 0.0.0.0/0.0.0.0:11222
2016-07-21 04:29:03,251 [myid:] - INFO [main:FileSnap@83] - Reading snapshot /home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/trunk/build/test/tmp/test393292139818745013.junit.dir/version-2/snapshot.2
2016-07-21 04:29:03,253 [myid:] - INFO [main:FileTxnSnapLog@298] - Snapshotting: 0x5 to /home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/trunk/build/test/tmp/test393292139818745013.junit.dir/version-2/snapshot.5
2016-07-21 04:29:03,255 [myid:] - INFO [main:FourLetterWordMain@85] - connecting to 127.0.0.1 11222
2016-07-21 04:29:03,255 [myid:] - INFO [NIOServerCxnFactory.AcceptThread:0.0.0.0/0.0.0.0:11222:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:48677
2016-07-21 04:29:03,256 [myid:] - INFO [NIOWorkerThread-1:NIOServerCnxn@485] - Processing stat command from /127.0.0.1:48677
2016-07-21 04:29:03,256 [myid:] - INFO [NIOWorkerThread-1:StatCommand@49] - Stat command output
2016-07-21 04:29:03,256 [myid:] - INFO [NIOWorkerThread-1:NIOServerCnxn@607] - Closed socket connection for client /127.0.0.1:48677 (no session established for client)
2016-07-21 04:29:03,257 [myid:] - INFO [ConnnectionExpirer:NIOServerCnxnFactory$ConnectionExpirerThread@583] - ConnnectionExpirerThread interrupted
2016-07-21 04:29:03,257 [myid:] - INFO [NIOServerCxnFactory.SelectorThread-0:NIOServerCnxnFactory$SelectorThread@420] - selector thread exitted run method
2016-07-21 04:29:03,257 [myid:] - INFO [NIOServerCxnFactory.AcceptThread:0.0.0.0/0.0.0.0:11222:NIOServerCnxnFactory$AcceptThread@219] - accept thread exitted run method
2016-07-21 04:29:03,257 [myid:] - INFO [main:ZooKeeperServer@498] - shutting down
2016-07-21 04:29:03,257 [myid:] - INFO [main:SessionTrackerImpl@232] - Shutting down
2016-07-21 04:29:03,258 [myid:] - INFO [main:PrepRequestProcessor@965] - Shutting down
2016-07-21 04:29:03,258 [myid:] - INFO [main:SyncRequestProcessor@191] - Shutting down
2016-07-21 04:29:03,258 [myid:] - INFO [ProcessThread(sid:0 cport:11222)::PrepRequestProcessor@154] - PrepRequestProcessor exited loop!
2016-07-21 04:29:03,258 [myid:] - INFO [SyncThread:0:SyncRequestProcessor@169] - SyncRequestProcessor exited!
2016-07-21 04:29:03,258 [myid:] - INFO [main:FinalRequestProcessor@479] - shutdown of request processor complete
2016-07-21 04:29:03,259 [myid:] - INFO [main:MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=StandaloneServer_port11222,name1=InMemoryDataTree]
2016-07-21 04:29:03,259 [myid:] - INFO [main:MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=StandaloneServer_port11222]
2016-07-21 04:29:03,259 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@82] - Memory used 8501
2016-07-21 04:29:03,259 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@87] - Number of threads 8
2016-07-21 04:29:03,260 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@102] - FINISHED TEST METHOD testReloadSnapshotWithMissingParent
2016-07-21 04:29:03,260 [myid:] - INFO [main:ZKTestCase$1@65] - SUCCEEDED testReloadSnapshotWithMissingParent
2016-07-21 04:29:03,260 [myid:] - INFO [main:ZKTestCase$1@60] - FINISHED testReloadSnapshotWithMissingParent
2016-07-21 04:29:03,262 [myid:] - INFO [main:ZKTestCase$1@55] - STARTING testRestoreWithTransactionErrors
2016-07-21 04:29:03,262 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@77] - RUNNING TEST METHOD testRestoreWithTransactionErrors
2016-07-21 04:29:03,262 [myid:] - INFO [main:ZooKeeperServer@858] - minSessionTimeout set to 6000
2016-07-21 04:29:03,263 [myid:] - INFO [main:ZooKeeperServer@867] - maxSessionTimeout set to 60000
2016-07-21 04:29:03,263 [myid:] - INFO [main:ZooKeeperServer@156] - Created server with tickTime 3000 minSessionTimeout 6000 maxSessionTimeout 60000 datadir /home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/trunk/build/test/tmp/test7440274560972421200.junit.dir/version-2 snapdir /home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/trunk/build/test/tmp/test7440274560972421200.junit.dir/version-2
2016-07-21 04:29:03,263 [myid:] - INFO [main:NIOServerCnxnFactory@673] - Configuring NIO connection handler with 10s sessionless connection timeout, 1 selector thread(s), 8 worker threads, and 64 kB direct buffers.
2016-07-21 04:29:03,263 [myid:] - INFO [main:NIOServerCnxnFactory@686] - binding to port 0.0.0.0/0.0.0.0:11222
2016-07-21 04:29:03,264 [myid:] - INFO [main:FileTxnSnapLog@298] - Snapshotting: 0x0 to /home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/trunk/build/test/tmp/test7440274560972421200.junit.dir/version-2/snapshot.0
2016-07-21 04:29:03,265 [myid:] - INFO [main:FourLetterWordMain@85] - connecting to 127.0.0.1 11222
2016-07-21 04:29:03,266 [myid:] - INFO [NIOServerCxnFactory.AcceptThread:0.0.0.0/0.0.0.0:11222:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:48678
2016-07-21 04:29:03,267 [myid:] - INFO [NIOWorkerThread-1:NIOServerCnxn@485] - Processing stat command from /127.0.0.1:48678
2016-07-21 04:29:03,267 [myid:] - INFO [NIOWorkerThread-1:StatCommand@49] - Stat command output
2016-07-21 04:29:03,267 [myid:] - INFO [NIOWorkerThread-1:NIOServerCnxn@607] - Closed socket connection for client /127.0.0.1:48678 (no session established for client)
2016-07-21 04:29:03,268 [myid:] - INFO [main:ZooKeeper@855] - Initiating client connection, connectString=127.0.0.1:11222 sessionTimeout=3000 watcher=org.apache.zookeeper.test.LoadFromLogTest@5763013f
2016-07-21 04:29:03,268 [myid:127.0.0.1:11222] - INFO [main-SendThread(127.0.0.1:11222):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:11222. Will not attempt to authenticate using SASL (unknown error)
2016-07-21 04:29:03,269 [myid:127.0.0.1:11222] - INFO [main-SendThread(127.0.0.1:11222):ClientCnxn$SendThread@948] - Socket connection established, initiating session, client: /127.0.0.1:48679, server: 127.0.0.1/127.0.0.1:11222
2016-07-21 04:29:03,269 [myid:] - INFO [NIOServerCxnFactory.AcceptThread:0.0.0.0/0.0.0.0:11222:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:48679
2016-07-21 04:29:03,270 [myid:] - INFO [NIOWorkerThread-2:ZooKeeperServer@964] - Client attempting to establish new session at /127.0.0.1:48679
2016-07-21 04:29:03,270 [myid:] - INFO [SyncThread:0:FileTxnLog@204] - Creating new log file: log.1
2016-07-21 04:29:03,273 [myid:] - INFO [SyncThread:0:ZooKeeperServer@678] - Established session 0x10002e291910000 with negotiated timeout 6000 for client /127.0.0.1:48679
2016-07-21 04:29:03,274 [myid:127.0.0.1:11222] - INFO [main-SendThread(127.0.0.1:11222):ClientCnxn$SendThread@1381] - Session establishment complete on server 127.0.0.1/127.0.0.1:11222, sessionid = 0x10002e291910000, negotiated timeout = 6000
2016-07-21 04:29:03,472 [myid:] - INFO [ProcessThread(sid:0 cport:11222)::PrepRequestProcessor@841] - Got user-level KeeperException when processing sessionid:0x10002e291910000 type:create cxid:0x1 zxid:0x2 txntype:-1 reqpath:n/a Error Path:/invaliddir Error:KeeperErrorCode = NoNode for /invaliddir
2016-07-21 04:29:03,475 [myid:] - INFO [ProcessThread(sid:0 cport:11222)::PrepRequestProcessor@841] - Got user-level KeeperException when processing sessionid:0x10002e291910000 type:create cxid:0x2 zxid:0x3 txntype:-1 reqpath:n/a Error Path:/invaliddir Error:KeeperErrorCode = NoNode for /invaliddir
2016-07-21 04:29:03,477 [myid:] - INFO [ProcessThread(sid:0 cport:11222)::PrepRequestProcessor@841] - Got user-level KeeperException when processing sessionid:0x10002e291910000 type:create cxid:0x3 zxid:0x4 txntype:-1 reqpath:n/a Error Path:/invaliddir Error:KeeperErrorCode = NoNode for /invaliddir
2016-07-21 04:29:03,479 [myid:] - INFO [ProcessThread(sid:0 cport:11222)::PrepRequestProcessor@841] - Got user-level KeeperException when processing sessionid:0x10002e291910000 type:create cxid:0x4 zxid:0x5 txntype:-1 reqpath:n/a Error Path:/invaliddir Error:KeeperErrorCode = NoNode for /invaliddir
2016-07-21 04:29:03,481 [myid:] - INFO [ProcessThread(sid:0 cport:11222)::PrepRequestProcessor@841] - Got user-level KeeperException when processing sessionid:0x10002e291910000 type:create cxid:0x5 zxid:0x6 txntype:-1 reqpath:n/a Error Path:/invaliddir Error:KeeperErrorCode = NoNode for /invaliddir
2016-07-21 04:29:03,483 [myid:] - INFO [ProcessThread(sid:0 cport:11222)::PrepRequestProcessor@841] - Got user-level KeeperException when processing sessionid:0x10002e291910000 type:create cxid:0x6 zxid:0x7 txntype:-1 reqpath:n/a Error Path:/invaliddir Error:KeeperErrorCode = NoNode for /invaliddir
2016-07-21 04:29:03,485 [myid:] - INFO [ProcessThread(sid:0 cport:11222)::PrepRequestProcessor@841] - Got user-level KeeperException when processing sessionid:0x10002e291910000 type:create cxid:0x7 zxid:0x8 txntype:-1 reqpath:n/a Error Path:/invaliddir Error:KeeperErrorCode = NoNode for /invaliddir
2016-07-21 04:29:03,487 [myid:] - INFO [ProcessThread(sid:0 cport:11222)::PrepRequestProcessor@841] - Got user-level KeeperException when processing sessionid:0x10002e291910000 type:create cxid:0x8 zxid:0x9 txntype:-1 reqpath:n/a Error Path:/invaliddir Error:KeeperErrorCode = NoNode for /invaliddir
2016-07-21 04:29:03,489 [myid:] - INFO [ProcessThread(sid:0 cport:11222)::PrepRequestProcessor@841] - Got user-level KeeperException when processing sessionid:0x10002e291910000 type:create cxid:0x9 zxid:0xa txntype:-1 reqpath:n/a Error Path:/invaliddir Error:KeeperErrorCode = NoNode for /invaliddir
2016-07-21 04:29:03,491 [myid:] - INFO [ProcessThread(sid:0 cport:11222)::PrepRequestProcessor@841] - Got user-level KeeperException when processing sessionid:0x10002e291910000 type:create cxid:0xa zxid:0xb txntype:-1 reqpath:n/a Error Path:/invaliddir Error:KeeperErrorCode = NoNode for /invaliddir
2016-07-21 04:29:03,492 [myid:] - INFO [ProcessThread(sid:0 cport:11222)::PrepRequestProcessor@841] - Got user-level KeeperException when processing sessionid:0x10002e291910000 type:create cxid:0xb zxid:0xc txntype:-1 reqpath:n/a Error Path:/invaliddir Error:KeeperErrorCode = NoNode for /invaliddir
2016-07-21 04:29:03,494 [myid:] - INFO [ProcessThread(sid:0 cport:11222)::PrepRequestProcessor@841] - Got user-level KeeperException when processing sessionid:0x10002e291910000 type:create cxid:0xc zxid:0xd txntype:-1 reqpath:n/a Error Path:/invaliddir Error:KeeperErrorCode = NoNode for /invaliddir
2016-07-21 04:29:03,496 [myid:] - INFO [ProcessThread(sid:0 cport:11222)::PrepRequestProcessor@841] - Got user-level KeeperException when processing sessionid:0x10002e291910000 type:create cxid:0xd zxid:0xe txntype:-1 reqpath:n/a Error Path:/invaliddir Error:KeeperErrorCode = NoNode for /invaliddir
2016-07-21 04:29:03,498 [myid:] - INFO [ProcessThread(sid:0 cport:11222)::PrepRequestProcessor@841] - Got user-level KeeperException when processing sessionid:0x10002e291910000 type:create cxid:0xe zxid:0xf txntype:-1 reqpath:n/a Error Path:/invaliddir Error:KeeperErrorCode = NoNode for /invaliddir
2016-07-21 04:29:03,499 [myid:] - INFO [ProcessThread(sid:0 cport:11222)::PrepRequestProcessor@841] - Got user-level KeeperException when processing sessionid:0x10002e291910000 type:create cxid:0xf zxid:0x10 txntype:-1 reqpath:n/a Error Path:/invaliddir Error:KeeperErrorCode = NoNode for /invaliddir
2016-07-21 04:29:03,501 [myid:] - INFO [ProcessThread(sid:0 cport:11222)::PrepRequestProcessor@841] - Got user-level KeeperException when processing sessionid:0x10002e291910000 type:create cxid:0x10 zxid:0x11 txntype:-1 reqpath:n/a Error Path:/invaliddir Error:KeeperErrorCode = NoNode for /invaliddir
2016-07-21 04:29:03,503 [myid:] - INFO [ProcessThread(sid:0 cport:11222)::PrepRequestProcessor@841] - Got user-level KeeperException when processing sessionid:0x10002e291910000 type:create cxid:0x11 zxid:0x12 txntype:-1 reqpath:n/a Error Path:/invaliddir Error:KeeperErrorCode = NoNode for /invaliddir
2016-07-21 04:29:03,505 [myid:] - INFO [ProcessThread(sid:0 cport:11222)::PrepRequestProcessor@841] - Got user-level KeeperException when processing sessionid:0x10002e291910000 type:create cxid:0x12 zxid:0x13 txntype:-1 reqpath:n/a Error Path:/invaliddir Error:KeeperErrorCode = NoNode for /invaliddir
2016-07-21 04:29:03,507 [myid:] - INFO [ProcessThread(sid:0 cport:11222)::PrepRequestProcessor@841] - Got user-level KeeperException when processing sessionid:0x10002e291910000 type:create cxid:0x13 zxid:0x14 txntype:-1 reqpath:n/a Error Path:/invaliddir Error:KeeperErrorCode = NoNode for /invaliddir
2016-07-21 04:29:03,508 [myid:] - INFO [ProcessThread(sid:0 cport:11222)::PrepRequestProcessor@841] - Got user-level KeeperException when processing sessionid:0x10002e291910000 type:create cxid:0x14 zxid:0x15 txntype:-1 reqpath:n/a Error Path:/invaliddir Error:KeeperErrorCode = NoNode for /invaliddir
2016-07-21 04:29:03,510 [myid:] - INFO [ProcessThread(sid:0 cport:11222)::PrepRequestProcessor@841] - Got user-level KeeperException when processing sessionid:0x10002e291910000 type:create cxid:0x15 zxid:0x16 txntype:-1 reqpath:n/a Error Path:/invaliddir Error:KeeperErrorCode = NoNode for /invaliddir
2016-07-21 04:29:03,512 [myid:] - INFO [ProcessThread(sid:0 cport:11222)::PrepRequestProcessor@841] - Got user-level KeeperException when processing sessionid:0x10002e291910000 type:create cxid:0x16 zxid:0x17 txntype:-1 reqpath:n/a Error Path:/invaliddir Error:KeeperErrorCode = NoNode for /invaliddir
2016-07-21 04:29:03,514 [myid:] - INFO [ProcessThread(sid:0 cport:11222)::PrepRequestProcessor@841] - Got user-level KeeperException when processing sessionid:0x10002e291910000 type:create cxid:0x17 zxid:0x18 txntype:-1 reqpath:n/a Error Path:/invaliddir Error:KeeperErrorCode = NoNode for /invaliddir
2016-07-21 04:29:03,516 [myid:] - INFO [ProcessThread(sid:0 cport:11222)::PrepRequestProcessor@841] - Got user-level KeeperException when processing sessionid:0x10002e291910000 type:create cxid:0x18 zxid:0x19 txntype:-1 reqpath:n/a Error Path:/invaliddir Error:KeeperErrorCode = NoNode for /invaliddir
2016-07-21 04:29:03,517 [myid:] - INFO [ProcessThread(sid:0 cport:11222)::PrepRequestProcessor@841] - Got user-level KeeperException when processing sessionid:0x10002e291910000 type:create cxid:0x19 zxid:0x1a txntype:-1 reqpath:n/a Error Path:/invaliddir Error:KeeperErrorCode = NoNode for /invaliddir
2016-07-21 04:29:03,519 [myid:] - INFO [ProcessThread(sid:0 cport:11222)::PrepRequestProcessor@841] - Got user-level KeeperException when processing sessionid:0x10002e291910000 type:create cxid:0x1a zxid:0x1b txntype:-1 reqpath:n/a Error Path:/invaliddir Error:KeeperErrorCode = NoNode for /invaliddir
2016-07-21 04:29:03,521 [myid:] - INFO [ProcessThread(sid:0 cport:11222)::PrepRequestProcessor@841] - Got user-level KeeperException when processing sessionid:0x10002e291910000 type:create cxid:0x1b zxid:0x1c txntype:-1 reqpath:n/a Error Path:/invaliddir Error:KeeperErrorCode = NoNode for /invaliddir
2016-07-21 04:29:03,523 [myid:] - INFO [ProcessThread(sid:0 cport:11222)::PrepRequestProcessor@841] - Got user-level KeeperException when processing sessionid:0x10002e291910000 type:create cxid:0x1c zxid:0x1d txntype:-1 reqpath:n/a Error Path:/invaliddir Error:KeeperErrorCode = NoNode for /invaliddir
2016-07-21 04:29:03,524 [myid:] - INFO [ProcessThread(sid:0 cport:11222)::PrepRequestProcessor@841] - Got user-level KeeperException when processing sessionid:0x10002e291910000 type:create cxid:0x1d zxid:0x1e txntype:-1 reqpath:n/a Error Path:/invaliddir Error:KeeperErrorCode = NoNode for /invaliddir
2016-07-21 04:29:03,526 [myid:] - INFO [ProcessThread(sid:0 cport:11222)::PrepRequestProcessor@841] - Got user-level KeeperException when processing sessionid:0x10002e291910000 type:create cxid:0x1e zxid:0x1f txntype:-1 reqpath:n/a Error Path:/invaliddir Error:KeeperErrorCode = NoNode for /invaliddir
2016-07-21 04:29:03,528 [myid:] - INFO [ProcessThread(sid:0 cport:11222)::PrepRequestProcessor@841] - Got user-level KeeperException when processing sessionid:0x10002e291910000 type:create cxid:0x1f zxid:0x20 txntype:-1 reqpath:n/a Error Path:/invaliddir Error:KeeperErrorCode = NoNode for /invaliddir
2016-07-21 04:29:03,530 [myid:] - INFO [ProcessThread(sid:0 cport:11222)::PrepRequestProcessor@841] - Got user-level KeeperException when processing sessionid:0x10002e291910000 type:create cxid:0x20 zxid:0x21 txntype:-1 reqpath:n/a Error Path:/invaliddir Error:KeeperErrorCode = NoNode for /invaliddir
2016-07-21 04:29:03,531 [myid:] - INFO [ProcessThread(sid:0 cport:11222)::PrepRequestProcessor@841] - Got user-level KeeperException when processing sessionid:0x10002e291910000 type:create cxid:0x21 zxid:0x22 txntype:-1 reqpath:n/a Error Path:/invaliddir Error:KeeperErrorCode = NoNode for /invaliddir
2016-07-21 04:29:03,534 [myid:] - INFO [ProcessThread(sid:0 cport:11222)::PrepRequestProcessor@841] - Got user-level KeeperException when processing sessionid:0x10002e291910000 type:create cxid:0x22 zxid:0x23 txntype:-1 reqpath:n/a Error Path:/invaliddir Error:KeeperErrorCode = NoNode for /invaliddir
2016-07-21 04:29:03,535 [myid:] - INFO [ProcessThread(sid:0 cport:11222)::PrepRequestProcessor@841] - Got user-level KeeperException when processing sessionid:0x10002e291910000 type:create cxid:0x23 zxid:0x24 txntype:-1 reqpath:n/a Error Path:/invaliddir Error:KeeperErrorCode = NoNode for /invaliddir
2016-07-21 04:29:03,537 [myid:] - INFO [ProcessThread(sid:0 cport:11222)::PrepRequestProcessor@841] - Got user-level KeeperException when processing sessionid:0x10002e291910000 type:create cxid:0x24 zxid:0x25 txntype:-1 reqpath:n/a Error Path:/invaliddir Error:KeeperErrorCode = NoNode for /invaliddir
2016-07-21 04:29:03,539 [myid:] - INFO [ProcessThread(sid:0 cport:11222)::PrepRequestProcessor@841] - Got user-level KeeperException when processing sessionid:0x10002e291910000 type:create cxid:0x25 zxid:0x26 txntype:-1 reqpath:n/a Error Path:/invaliddir Error:KeeperErrorCode = NoNode for /invaliddir
2016-07-21 04:29:03,541 [myid:] - INFO [ProcessThread(sid:0 cport:11222)::PrepRequestProcessor@841] - Got user-level KeeperException when processing sessionid:0x10002e291910000 type:create cxid:0x26 zxid:0x27 txntype:-1 reqpath:n/a Error Path:/invaliddir Error:KeeperErrorCode = NoNode for /invaliddir
2016-07-21 04:29:03,542 [myid:] - INFO [ProcessThread(sid:0 cport:11222)::PrepRequestProcessor@841] - Got user-level KeeperException when processing sessionid:0x10002e291910000 type:create cxid:0x27 zxid:0x28 txntype:-1 reqpath:n/a Error Path:/invaliddir Error:KeeperErrorCode = NoNode for /invaliddir
2016-07-21 04:29:03,545 [myid:] - INFO [ProcessThread(sid:0 cport:11222)::PrepRequestProcessor@841] - Got user-level KeeperException when processing sessionid:0x10002e291910000 type:create cxid:0x28 zxid:0x29 txntype:-1 reqpath:n/a Error Path:/invaliddir Error:KeeperErrorCode = NoNode for /invaliddir
2016-07-21 04:29:03,548 [myid:] - INFO [ProcessThread(sid:0 cport:11222)::PrepRequestProcessor@841] - Got user-level KeeperException when processing sessionid:0x10002e291910000 type:create cxid:0x29 zxid:0x2a txntype:-1 reqpath:n/a Error Path:/invaliddir Error:KeeperErrorCode = NoNode for /invaliddir
2016-07-21 04:29:03,549 [myid:] - INFO [ProcessThread(sid:0 cport:11222)::PrepRequestProcessor@841] - Got user-level KeeperException when processing sessionid:0x10002e291910000 type:create cxid:0x2a zxid:0x2b txntype:-1 reqpath:n/a Error Path:/invaliddir Error:KeeperErrorCode = NoNode for /invaliddir
2016-07-21 04:29:03,551 [myid:] - INFO [ProcessThread(sid:0 cport:11222)::PrepRequestProcessor@841] - Got user-level KeeperException when processing sessionid:0x10002e291910000 type:create cxid:0x2b zxid:0x2c txntype:-1 reqpath:n/a Error Path:/invaliddir Error:KeeperErrorCode = NoNode for /invaliddir
2016-07-21 04:29:03,552 [myid:] - INFO [ProcessThread(sid:0 cport:11222)::PrepRequestProcessor@841] - Got user-level KeeperException when processing sessionid:0x10002e291910000 type:create cxid:0x2c zxid:0x2d txntype:-1 reqpath:n/a Error Path:/invaliddir Error:KeeperErrorCode = NoNode for /invaliddir
2016-07-21 04:29:03,554 [myid:] - INFO [ProcessThread(sid:0 cport:11222)::PrepRequestProcessor@841] - Got user-level KeeperException when processing sessionid:0x10002e291910000 type:create cxid:0x2d zxid:0x2e txntype:-1 reqpath:n/a Error Path:/invaliddir Error:KeeperErrorCode = NoNode for /invaliddir
2016-07-21 04:29:03,556 [myid:] - INFO [ProcessThread(sid:0 cport:11222)::PrepRequestProcessor@841] - Got user-level KeeperException when processing sessionid:0x10002e291910000 type:create cxid:0x2e zxid:0x2f txntype:-1 reqpath:n/a Error Path:/invaliddir Error:KeeperErrorCode = NoNode for /invaliddir
2016-07-21 04:29:03,557 [myid:] - INFO [ProcessThread(sid:0 cport:11222)::PrepRequestProcessor@841] - Got user-level KeeperException when processing sessionid:0x10002e291910000 type:create cxid:0x2f zxid:0x30 txntype:-1 reqpath:n/a Error Path:/invaliddir Error:KeeperErrorCode = NoNode for /invaliddir
2016-07-21 04:29:03,559 [myid:] - INFO [ProcessThread(sid:0 cport:11222)::PrepRequestProcessor@841] - Got user-level KeeperException when processing sessionid:0x10002e291910000 type:create cxid:0x30 zxid:0x31 txntype:-1 reqpath:n/a Error Path:/invaliddir Error:KeeperErrorCode = NoNode for /invaliddir
2016-07-21 04:29:03,561 [myid:] - INFO [ProcessThread(sid:0 cport:11222)::PrepRequestProcessor@841] - Got user-level KeeperException when processing sessionid:0x10002e291910000 type:create cxid:0x31 zxid:0x32 txntype:-1 reqpath:n/a Error Path:/invaliddir Error:KeeperErrorCode = NoNode for /invaliddir
2016-07-21 04:29:03,563 [myid:] - INFO [ProcessThread(sid:0 cport:11222)::PrepRequestProcessor@841] - Got user-level KeeperException when processing sessionid:0x10002e291910000 type:create cxid:0x32 zxid:0x33 txntype:-1 reqpath:n/a Error Path:/invaliddir Error:KeeperErrorCode = NoNode for /invaliddir
2016-07-21 04:29:03,564 [myid:] - INFO [ProcessThread(sid:0 cport:11222)::PrepRequestProcessor@841] - Got user-level KeeperException when processing sessionid:0x10002e291910000 type:create cxid:0x33 zxid:0x34 txntype:-1 reqpath:n/a Error Path:/invaliddir Error:KeeperErrorCode = NoNode for /invaliddir
2016-07-21 04:29:03,565 [myid:] - INFO [ProcessThread(sid:0 cport:11222)::PrepRequestProcessor@841] - Got user-level KeeperException when processing sessionid:0x10002e291910000 type:create cxid:0x34 zxid:0x35 txntype:-1 reqpath:n/a Error Path:/invaliddir Error:KeeperErrorCode = NoNode for /invaliddir
2016-07-21 04:29:03,567 [myid:] - INFO [ProcessThread(sid:0 cport:11222)::PrepRequestProcessor@841] - Got user-level KeeperException when processing sessionid:0x10002e291910000 type:create cxid:0x35 zxid:0x36 txntype:-1 reqpath:n/a Error Path:/invaliddir Error:KeeperErrorCode = NoNode for /invaliddir
2016-07-21 04:29:03,569 [myid:] - INFO [ProcessThread(sid:0 cport:11222)::PrepRequestProcessor@841] - Got user-level KeeperException when processing sessionid:0x10002e291910000 type:create cxid:0x36 zxid:0x37 txntype:-1 reqpath:n/a Error Path:/invaliddir Error:KeeperErrorCode = NoNode for /invaliddir
2016-07-21 04:29:03,571 [myid:] - INFO [ProcessThread(sid:0 cport:11222)::PrepRequestProcessor@841] - Got user-level KeeperException when processing sessionid:0x10002e291910000 type:create cxid:0x37 zxid:0x38 txntype:-1 reqpath:n/a Error Path:/invaliddir Error:KeeperErrorCode = NoNode for /invaliddir
2016-07-21 04:29:03,572 [myid:] - INFO [ProcessThread(sid:0 cport:11222)::PrepRequestProcessor@841] - Got user-level KeeperException when processing sessionid:0x10002e291910000 type:create cxid:0x38 zxid:0x39 txntype:-1 reqpath:n/a Error Path:/invaliddir Error:KeeperErrorCode = NoNode for /invaliddir
2016-07-21 04:29:03,574 [myid:] - INFO [ProcessThread(sid:0 cport:11222)::PrepRequestProcessor@841] - Got user-level KeeperException when processing sessionid:0x10002e291910000 type:create cxid:0x39 zxid:0x3a txntype:-1 reqpath:n/a Error Path:/invaliddir Error:KeeperErrorCode = NoNode for /invaliddir
2016-07-21 04:29:03,575 [myid:] - INFO [ProcessThread(sid:0 cport:11222)::PrepRequestProcessor@841] - Got user-level KeeperException when processing sessionid:0x10002e291910000 type:create cxid:0x3a zxid:0x3b txntype:-1 reqpath:n/a Error Path:/invaliddir Error:KeeperErrorCode = NoNode for /invaliddir
2016-07-21 04:29:03,577 [myid:] - INFO [ProcessThread(sid:0 cport:11222)::PrepRequestProcessor@841] - Got user-level KeeperException when processing sessionid:0x10002e291910000 type:create cxid:0x3b zxid:0x3c txntype:-1 reqpath:n/a Error Path:/invaliddir Error:KeeperErrorCode = NoNode for /invaliddir
2016-07-21 04:29:03,578 [myid:] - INFO [ProcessThread(sid:0 cport:11222)::PrepRequestProcessor@841] - Got user-level KeeperException when processing sessionid:0x10002e291910000 type:create cxid:0x3c zxid:0x3d txntype:-1 reqpath:n/a Error Path:/invaliddir Error:KeeperErrorCode = NoNode for /invaliddir
2016-07-21 04:29:03,580 [myid:] - INFO [ProcessThread(sid:0 cport:11222)::PrepRequestProcessor@841] - Got user-level KeeperException when processing sessionid:0x10002e291910000 type:create cxid:0x3d zxid:0x3e txntype:-1 reqpath:n/a Error Path:/invaliddir Error:KeeperErrorCode = NoNode for /invaliddir
2016-07-21 04:29:03,581 [myid:] - INFO [ProcessThread(sid:0 cport:11222)::PrepRequestProcessor@841] - Got user-level KeeperException when processing sessionid:0x10002e291910000 type:create cxid:0x3e zxid:0x3f txntype:-1 reqpath:n/a Error Path:/invaliddir Error:KeeperErrorCode = NoNode for /invaliddir
2016-07-21 04:29:03,583 [myid:] - INFO [ProcessThread(sid:0 cport:11222)::PrepRequestProcessor@841] - Got user-level KeeperException when processing sessionid:0x10002e291910000 type:create cxid:0x3f zxid:0x40 txntype:-1 reqpath:n/a Error Path:/invaliddir Error:KeeperErrorCode = NoNode for /invaliddir
2016-07-21 04:29:03,584 [myid:] - INFO [ProcessThread(sid:0 cport:11222)::PrepRequestProcessor@841] - Got user-level KeeperException when processing sessionid:0x10002e291910000 type:create cxid:0x40 zxid:0x41 txntype:-1 reqpath:n/a Error Path:/invaliddir Error:KeeperErrorCode = NoNode for /invaliddir
2016-07-21 04:29:03,586 [myid:] - INFO [ProcessThread(sid:0 cport:11222)::PrepRequestProcessor@841] - Got user-level KeeperException when processing sessionid:0x10002e291910000 type:create cxid:0x41 zxid:0x42 txntype:-1 reqpath:n/a Error Path:/invaliddir Error:KeeperErrorCode = NoNode for /invaliddir
2016-07-21 04:29:03,587 [myid:] - INFO [ProcessThread(sid:0 cport:11222)::PrepRequestProcessor@841] - Got user-level KeeperException when processing sessionid:0x10002e291910000 type:create cxid:0x42 zxid:0x43 txntype:-1 reqpath:n/a Error Path:/invaliddir Error:KeeperErrorCode = NoNode for /invaliddir
2016-07-21 04:29:03,589 [myid:] - INFO [ProcessThread(sid:0 cport:11222)::PrepRequestProcessor@841] - Got user-level KeeperException when processing sessionid:0x10002e291910000 type:create cxid:0x43 zxid:0x44 txntype:-1 reqpath:n/a Error Path:/invaliddir Error:KeeperErrorCode = NoNode for /invaliddir
2016-07-21 04:29:03,591 [myid:] - INFO [ProcessThread(sid:0 cport:11222)::PrepRequestProcessor@841] - Got user-level KeeperException when processing sessionid:0x10002e291910000 type:create cxid:0x44 zxid:0x45 txntype:-1 reqpath:n/a Error Path:/invaliddir Error:KeeperErrorCode = NoNode for /invaliddir
2016-07-21 04:29:03,592 [myid:] - INFO [ProcessThread(sid:0 cport:11222)::PrepRequestProcessor@841] - Got user-level KeeperException when processing sessionid:0x10002e291910000 type:create cxid:0x45 zxid:0x46 txntype:-1 reqpath:n/a Error Path:/invaliddir Error:KeeperErrorCode = NoNode for /invaliddir
2016-07-21 04:29:03,594 [myid:] - INFO [ProcessThread(sid:0 cport:11222)::PrepRequestProcessor@841] - Got user-level KeeperException when processing sessionid:0x10002e291910000 type:create cxid:0x46 zxid:0x47 txntype:-1 reqpath:n/a Error Path:/invaliddir Error:KeeperErrorCode = NoNode for /invaliddir
2016-07-21 04:29:03,595 [myid:] - INFO [ProcessThread(sid:0 cport:11222)::PrepRequestProcessor@841] - Got user-level KeeperException when processing sessionid:0x10002e291910000 type:create cxid:0x47 zxid:0x48 txntype:-1 reqpath:n/a Error Path:/invaliddir Error:KeeperErrorCode = NoNode for /invaliddir
2016-07-21 04:29:03,597 [myid:] - INFO [ProcessThread(sid:0 cport:11222)::PrepRequestProcessor@841] - Got user-level KeeperException when processing sessionid:0x10002e291910000 type:create cxid:0x48 zxid:0x49 txntype:-1 reqpath:n/a Error Path:/invaliddir Error:KeeperErrorCode = NoNode for /invaliddir
2016-07-21 04:29:03,598 [myid:] - INFO [ProcessThread(sid:0 cport:11222)::PrepRequestProcessor@841] - Got user-level KeeperException when processing sessionid:0x10002e291910000 type:create cxid:0x49 zxid:0x4a txntype:-1 reqpath:n/a Error Path:/invaliddir Error:KeeperErrorCode = NoNode for /invaliddir
2016-07-21 04:29:03,600 [myid:] - INFO [ProcessThread(sid:0 cport:11222)::PrepRequestProcessor@841] - Got user-level KeeperException when processing sessionid:0x10002e291910000 type:create cxid:0x4a zxid:0x4b txntype:-1 reqpath:n/a Error Path:/invaliddir Error:KeeperErrorCode = NoNode for /invaliddir
2016-07-21 04:29:03,602 [myid:] - INFO [ProcessThread(sid:0 cport:11222)::PrepRequestProcessor@841] - Got user-level KeeperException when processing sessionid:0x10002e291910000 type:create cxid:0x4b zxid:0x4c txntype:-1 reqpath:n/a Error Path:/invaliddir Error:KeeperErrorCode = NoNode for /invaliddir
2016-07-21 04:29:03,603 [myid:] - INFO [ProcessThread(sid:0 cport:11222)::PrepRequestProcessor@841] - Got user-level KeeperException when processing sessionid:0x10002e291910000 type:create cxid:0x4c zxid:0x4d txntype:-1 reqpath:n/a Error Path:/invaliddir Error:KeeperErrorCode = NoNode for /invaliddir
2016-07-21 04:29:03,605 [myid:] - INFO [ProcessThread(sid:0 cport:11222)::PrepRequestProcessor@841] - Got user-level KeeperException when processing sessionid:0x10002e291910000 type:create cxid:0x4d zxid:0x4e txntype:-1 reqpath:n/a Error Path:/invaliddir Error:KeeperErrorCode = NoNode for /invaliddir
2016-07-21 04:29:03,606 [myid:] - INFO [ProcessThread(sid:0 cport:11222)::PrepRequestProcessor@841] - Got user-level KeeperException when processing sessionid:0x10002e291910000 type:create cxid:0x4e zxid:0x4f txntype:-1 reqpath:n/a Error Path:/invaliddir Error:KeeperErrorCode = NoNode for /invaliddir
2016-07-21 04:29:03,608 [myid:] - INFO [ProcessThread(sid:0 cport:11222)::PrepRequestProcessor@841] - Got user-level KeeperException when processing sessionid:0x10002e291910000 type:create cxid:0x4f zxid:0x50 txntype:-1 reqpath:n/a Error Path:/invaliddir Error:KeeperErrorCode = NoNode for /invaliddir
2016-07-21 04:29:03,610 [myid:] - INFO [ProcessThread(sid:0 cport:11222)::PrepRequestProcessor@841] - Got user-level KeeperException when processing sessionid:0x10002e291910000 type:create cxid:0x50 zxid:0x51 txntype:-1 reqpath:n/a Error Path:/invaliddir Error:KeeperErrorCode = NoNode for /invaliddir
2016-07-21 04:29:03,611 [myid:] - INFO [ProcessThread(sid:0 cport:11222)::PrepRequestProcessor@841] - Got user-level KeeperException when processing sessionid:0x10002e291910000 type:create cxid:0x51 zxid:0x52 txntype:-1 reqpath:n/a Error Path:/invaliddir Error:KeeperErrorCode = NoNode for /invaliddir
2016-07-21 04:29:03,613 [myid:] - INFO [ProcessThread(sid:0 cport:11222)::PrepRequestProcessor@841] - Got user-level KeeperException when processing sessionid:0x10002e291910000 type:create cxid:0x52 zxid:0x53 txntype:-1 reqpath:n/a Error Path:/invaliddir Error:KeeperErrorCode = NoNode for /invaliddir
2016-07-21 04:29:03,614 [myid:] - INFO [ProcessThread(sid:0 cport:11222)::PrepRequestProcessor@841] - Got user-level KeeperException when processing sessionid:0x10002e291910000 type:create cxid:0x53 zxid:0x54 txntype:-1 reqpath:n/a Error Path:/invaliddir Error:KeeperErrorCode = NoNode for /invaliddir
2016-07-21 04:29:03,617 [myid:] - INFO [ProcessThread(sid:0 cport:11222)::PrepRequestProcessor@841] - Got user-level KeeperException when processing sessionid:0x10002e291910000 type:create cxid:0x54 zxid:0x55 txntype:-1 reqpath:n/a Error
...[truncated 64722 chars]...
INFO [ProcessThread(sid:0 cport:11222)::PrepRequestProcessor@841] - Got user-level KeeperException when processing sessionid:0x10002e291910000 type:create cxid:0x12a zxid:0x12b txntype:-1 reqpath:n/a Error Path:/invaliddir Error:KeeperErrorCode = NoNode for /invaliddir
2016-07-21 04:29:03,976 [myid:] - INFO [ProcessThread(sid:0 cport:11222)::PrepRequestProcessor@841] - Got user-level KeeperException when processing sessionid:0x10002e291910000 type:create cxid:0x12b zxid:0x12c txntype:-1 reqpath:n/a Error Path:/invaliddir Error:KeeperErrorCode = NoNode for /invaliddir
2016-07-21 04:29:03,978 [myid:] - INFO [ProcessThread(sid:0 cport:11222)::PrepRequestProcessor@841] - Got user-level KeeperException when processing sessionid:0x10002e291910000 type:create cxid:0x12c zxid:0x12d txntype:-1 reqpath:n/a Error Path:/invaliddir Error:KeeperErrorCode = NoNode for /invaliddir
2016-07-21 04:29:03,980 [myid:] - INFO [ProcessThread(sid:0 cport:11222)::PrepRequestProcessor@647] - Processed session termination for sessionid: 0x10002e291910000
2016-07-21 04:29:03,981 [myid:] - INFO [NIOWorkerThread-5:MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=StandaloneServer_port11222,name1=Connections,name2=127.0.0.1,name3=0x10002e291910000]
2016-07-21 04:29:03,981 [myid:] - INFO [main-EventThread:ClientCnxn$EventThread@513] - EventThread shut down for session: 0x10002e291910000
2016-07-21 04:29:03,981 [myid:] - INFO [main:ZooKeeper@1313] - Session: 0x10002e291910000 closed
2016-07-21 04:29:03,981 [myid:] - INFO [NIOWorkerThread-5:NIOServerCnxn@607] - Closed socket connection for client /127.0.0.1:48679 which had sessionid 0x10002e291910000
2016-07-21 04:29:04,578 [myid:] - INFO [main:LoadFromLogTest@465] - Set lastProcessedZxid to 292
2016-07-21 04:29:04,579 [myid:] - INFO [main:FileTxnSnapLog@298] - Snapshotting: 0x124 to /home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/trunk/build/test/tmp/test7440274560972421200.junit.dir/version-2/snapshot.124
2016-07-21 04:29:04,579 [myid:] - INFO [main:ZooKeeperServer@498] - shutting down
2016-07-21 04:29:04,580 [myid:] - INFO [main:SessionTrackerImpl@232] - Shutting down
2016-07-21 04:29:04,580 [myid:] - INFO [main:PrepRequestProcessor@965] - Shutting down
2016-07-21 04:29:04,580 [myid:] - INFO [main:SyncRequestProcessor@191] - Shutting down
2016-07-21 04:29:04,580 [myid:] - INFO [ProcessThread(sid:0 cport:11222)::PrepRequestProcessor@154] - PrepRequestProcessor exited loop!
2016-07-21 04:29:04,580 [myid:] - INFO [SyncThread:0:SyncRequestProcessor@169] - SyncRequestProcessor exited!
2016-07-21 04:29:04,580 [myid:] - INFO [main:FinalRequestProcessor@479] - shutdown of request processor complete
2016-07-21 04:29:04,581 [myid:] - INFO [main:MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=StandaloneServer_port11222,name1=InMemoryDataTree]
2016-07-21 04:29:04,581 [myid:] - INFO [main:MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=StandaloneServer_port11222]
2016-07-21 04:29:04,582 [myid:] - INFO [ConnnectionExpirer:NIOServerCnxnFactory$ConnectionExpirerThread@583] - ConnnectionExpirerThread interrupted
2016-07-21 04:29:04,582 [myid:] - INFO [NIOServerCxnFactory.SelectorThread-0:NIOServerCnxnFactory$SelectorThread@420] - selector thread exitted run method
2016-07-21 04:29:04,583 [myid:] - INFO [NIOServerCxnFactory.AcceptThread:0.0.0.0/0.0.0.0:11222:NIOServerCnxnFactory$AcceptThread@219] - accept thread exitted run method
2016-07-21 04:29:04,583 [myid:] - INFO [main:ZooKeeperServer@858] - minSessionTimeout set to 6000
2016-07-21 04:29:04,584 [myid:] - INFO [main:ZooKeeperServer@867] - maxSessionTimeout set to 60000
2016-07-21 04:29:04,584 [myid:] - INFO [main:ZooKeeperServer@156] - Created server with tickTime 3000 minSessionTimeout 6000 maxSessionTimeout 60000 datadir /home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/trunk/build/test/tmp/test7440274560972421200.junit.dir/version-2 snapdir /home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/trunk/build/test/tmp/test7440274560972421200.junit.dir/version-2
2016-07-21 04:29:04,584 [myid:] - INFO [main:NIOServerCnxnFactory@673] - Configuring NIO connection handler with 10s sessionless connection timeout, 1 selector thread(s), 8 worker threads, and 64 kB direct buffers.
2016-07-21 04:29:04,584 [myid:] - INFO [main:NIOServerCnxnFactory@686] - binding to port 0.0.0.0/0.0.0.0:11222
2016-07-21 04:29:04,586 [myid:] - INFO [main:FileSnap@83] - Reading snapshot /home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/trunk/build/test/tmp/test7440274560972421200.junit.dir/version-2/snapshot.124
2016-07-21 04:29:04,593 [myid:] - INFO [main:FileTxnSnapLog@298] - Snapshotting: 0x12e to /home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/trunk/build/test/tmp/test7440274560972421200.junit.dir/version-2/snapshot.12e
2016-07-21 04:29:04,594 [myid:] - INFO [main:FourLetterWordMain@85] - connecting to 127.0.0.1 11222
2016-07-21 04:29:04,595 [myid:] - INFO [NIOServerCxnFactory.AcceptThread:0.0.0.0/0.0.0.0:11222:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:48680
2016-07-21 04:29:04,596 [myid:] - INFO [NIOWorkerThread-1:NIOServerCnxn@485] - Processing stat command from /127.0.0.1:48680
2016-07-21 04:29:04,596 [myid:] - INFO [NIOWorkerThread-1:StatCommand@49] - Stat command output
2016-07-21 04:29:04,596 [myid:] - INFO [NIOWorkerThread-1:NIOServerCnxn@607] - Closed socket connection for client /127.0.0.1:48680 (no session established for client)
2016-07-21 04:29:04,597 [myid:] - INFO [ConnnectionExpirer:NIOServerCnxnFactory$ConnectionExpirerThread@583] - ConnnectionExpirerThread interrupted
2016-07-21 04:29:04,597 [myid:] - INFO [NIOServerCxnFactory.AcceptThread:0.0.0.0/0.0.0.0:11222:NIOServerCnxnFactory$AcceptThread@219] - accept thread exitted run method
2016-07-21 04:29:04,597 [myid:] - INFO [NIOServerCxnFactory.SelectorThread-0:NIOServerCnxnFactory$SelectorThread@420] - selector thread exitted run method
2016-07-21 04:29:04,598 [myid:] - INFO [main:ZooKeeperServer@498] - shutting down
2016-07-21 04:29:04,598 [myid:] - INFO [main:SessionTrackerImpl@232] - Shutting down
2016-07-21 04:29:04,598 [myid:] - INFO [main:PrepRequestProcessor@965] - Shutting down
2016-07-21 04:29:04,598 [myid:] - INFO [main:SyncRequestProcessor@191] - Shutting down
2016-07-21 04:29:04,598 [myid:] - INFO [ProcessThread(sid:0 cport:11222)::PrepRequestProcessor@154] - PrepRequestProcessor exited loop!
2016-07-21 04:29:04,598 [myid:] - INFO [SyncThread:0:SyncRequestProcessor@169] - SyncRequestProcessor exited!
2016-07-21 04:29:04,599 [myid:] - INFO [main:FinalRequestProcessor@479] - shutdown of request processor complete
2016-07-21 04:29:04,599 [myid:] - INFO [main:MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=StandaloneServer_port11222,name1=InMemoryDataTree]
2016-07-21 04:29:04,599 [myid:] - INFO [main:MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=StandaloneServer_port11222]
2016-07-21 04:29:04,599 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@82] - Memory used 16613
2016-07-21 04:29:04,600 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@87] - Number of threads 10
2016-07-21 04:29:04,600 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@102] - FINISHED TEST METHOD testRestoreWithTransactionErrors
2016-07-21 04:29:04,600 [myid:] - INFO [main:ZKTestCase$1@65] - SUCCEEDED testRestoreWithTransactionErrors
2016-07-21 04:29:04,600 [myid:] - INFO [main:ZKTestCase$1@60] - FINISHED testRestoreWithTransactionErrors
2016-07-21 04:29:04,601 [myid:] - INFO [main:ZKTestCase$1@55] - STARTING testPad
2016-07-21 04:29:04,601 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@77] - RUNNING TEST METHOD testPad
2016-07-21 04:29:04,601 [myid:] - INFO [main:FileTxnLog@204] - Creating new log file: log.123
2016-07-21 04:29:04,602 [myid:] - INFO [main:LoadFromLogTest@343] - Received magic : 1514884167 Expected : 1514884167
2016-07-21 04:29:04,602 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@82] - Memory used 16921
2016-07-21 04:29:04,602 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@87] - Number of threads 10
2016-07-21 04:29:04,602 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@102] - FINISHED TEST METHOD testPad
2016-07-21 04:29:04,602 [myid:] - INFO [main:ZKTestCase$1@65] - SUCCEEDED testPad
2016-07-21 04:29:04,603 [myid:] - INFO [main:ZKTestCase$1@60] - FINISHED testPad
2016-07-21 04:29:04,603 [myid:] - INFO [main:ZKTestCase$1@55] - STARTING testLoad
2016-07-21 04:29:04,603 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@77] - RUNNING TEST METHOD testLoad
2016-07-21 04:29:04,604 [myid:] - INFO [main:ZooKeeperServer@858] - minSessionTimeout set to 6000
2016-07-21 04:29:04,604 [myid:] - INFO [main:ZooKeeperServer@867] - maxSessionTimeout set to 60000
2016-07-21 04:29:04,604 [myid:] - INFO [main:ZooKeeperServer@156] - Created server with tickTime 3000 minSessionTimeout 6000 maxSessionTimeout 60000 datadir /home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/trunk/build/test/tmp/test6104680937889031731.junit.dir/version-2 snapdir /home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/trunk/build/test/tmp/test6104680937889031731.junit.dir/version-2
2016-07-21 04:29:04,604 [myid:] - INFO [main:NIOServerCnxnFactory@673] - Configuring NIO connection handler with 10s sessionless connection timeout, 1 selector thread(s), 8 worker threads, and 64 kB direct buffers.
2016-07-21 04:29:04,651 [myid:] - INFO [main:NIOServerCnxnFactory@686] - binding to port 0.0.0.0/0.0.0.0:11222
2016-07-21 04:29:04,652 [myid:] - INFO [main:FileTxnSnapLog@298] - Snapshotting: 0x0 to /home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/trunk/build/test/tmp/test6104680937889031731.junit.dir/version-2/snapshot.0
2016-07-21 04:29:04,654 [myid:] - INFO [main:FourLetterWordMain@85] - connecting to 127.0.0.1 11222
2016-07-21 04:29:04,654 [myid:] - INFO [NIOServerCxnFactory.AcceptThread:0.0.0.0/0.0.0.0:11222:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:48681
2016-07-21 04:29:04,655 [myid:] - INFO [NIOWorkerThread-1:NIOServerCnxn@485] - Processing stat command from /127.0.0.1:48681
2016-07-21 04:29:04,656 [myid:] - INFO [NIOWorkerThread-1:StatCommand@49] - Stat command output
2016-07-21 04:29:04,656 [myid:] - INFO [NIOWorkerThread-1:NIOServerCnxn@607] - Closed socket connection for client /127.0.0.1:48681 (no session established for client)
2016-07-21 04:29:04,656 [myid:] - INFO [main:ZooKeeper@855] - Initiating client connection, connectString=127.0.0.1:11222 sessionTimeout=3000 watcher=org.apache.zookeeper.test.LoadFromLogTest@24fa3073
2016-07-21 04:29:04,657 [myid:127.0.0.1:11222] - INFO [main-SendThread(127.0.0.1:11222):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:11222. Will not attempt to authenticate using SASL (unknown error)
2016-07-21 04:29:04,658 [myid:] - INFO [NIOServerCxnFactory.AcceptThread:0.0.0.0/0.0.0.0:11222:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:48682
2016-07-21 04:29:04,658 [myid:127.0.0.1:11222] - INFO [main-SendThread(127.0.0.1:11222):ClientCnxn$SendThread@948] - Socket connection established, initiating session, client: /127.0.0.1:48682, server: 127.0.0.1/127.0.0.1:11222
2016-07-21 04:29:04,659 [myid:] - INFO [NIOWorkerThread-2:ZooKeeperServer@964] - Client attempting to establish new session at /127.0.0.1:48682
2016-07-21 04:29:04,659 [myid:] - INFO [SyncThread:0:FileTxnLog@204] - Creating new log file: log.1
2016-07-21 04:29:04,663 [myid:] - INFO [SyncThread:0:ZooKeeperServer@678] - Established session 0x10002e296fd0000 with negotiated timeout 6000 for client /127.0.0.1:48682
2016-07-21 04:29:04,663 [myid:127.0.0.1:11222] - INFO [main-SendThread(127.0.0.1:11222):ClientCnxn$SendThread@1381] - Session establishment complete on server 127.0.0.1/127.0.0.1:11222, sessionid = 0x10002e296fd0000, negotiated timeout = 6000
2016-07-21 04:29:04,839 [myid:] - INFO [Snapshot Thread:FileTxnSnapLog@298] - Snapshotting: 0x63 to /home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/trunk/build/test/tmp/test6104680937889031731.junit.dir/version-2/snapshot.63
2016-07-21 04:29:04,840 [myid:] - INFO [SyncThread:0:FileTxnLog@204] - Creating new log file: log.65
2016-07-21 04:29:04,958 [myid:] - INFO [Snapshot Thread:FileTxnSnapLog@298] - Snapshotting: 0xac to /home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/trunk/build/test/tmp/test6104680937889031731.junit.dir/version-2/snapshot.ac
2016-07-21 04:29:04,959 [myid:] - INFO [SyncThread:0:FileTxnLog@204] - Creating new log file: log.ae
2016-07-21 04:29:05,068 [myid:127.0.0.1:11222] - INFO [main-SendThread(127.0.0.1:11222):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:11222. Will not attempt to authenticate using SASL (unknown error)
2016-07-21 04:29:05,069 [myid:] - INFO [NIOServerCxnFactory.AcceptThread:0.0.0.0/0.0.0.0:11222:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:48683
2016-07-21 04:29:05,069 [myid:127.0.0.1:11222] - INFO [main-SendThread(127.0.0.1:11222):ClientCnxn$SendThread@948] - Socket connection established, initiating session, client: /127.0.0.1:48683, server: 127.0.0.1/127.0.0.1:11222
2016-07-21 04:29:05,070 [myid:] - INFO [NIOWorkerThread-1:ZooKeeperServer@969] - Client attempting to renew session 0x10002e28feb0000 at /127.0.0.1:48683
2016-07-21 04:29:05,070 [myid:] - INFO [NIOWorkerThread-1:ZooKeeperServer@686] - Invalid session 0x10002e28feb0000 for client /127.0.0.1:48683, probably expired
2016-07-21 04:29:05,071 [myid:] - INFO [NIOWorkerThread-4:NIOServerCnxn@607] - Closed socket connection for client /127.0.0.1:48683 which had sessionid 0x10002e28feb0000
2016-07-21 04:29:05,071 [myid:] - INFO [main-EventThread:ClientCnxn$EventThread@513] - EventThread shut down for session: 0x10002e28feb0000
2016-07-21 04:29:05,071 [myid:127.0.0.1:11222] - WARN [main-SendThread(127.0.0.1:11222):ClientCnxn$SendThread@1367] - Unable to reconnect to ZooKeeper service, session 0x10002e28feb0000 has expired
2016-07-21 04:29:05,071 [myid:127.0.0.1:11222] - INFO [main-SendThread(127.0.0.1:11222):ClientCnxn$SendThread@1227] - Unable to reconnect to ZooKeeper service, session 0x10002e28feb0000 has expired, closing socket connection
2016-07-21 04:29:05,106 [myid:] - INFO [Snapshot Thread:FileTxnSnapLog@298] - Snapshotting: 0x107 to /home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/trunk/build/test/tmp/test6104680937889031731.junit.dir/version-2/snapshot.107
2016-07-21 04:29:05,107 [myid:] - INFO [SyncThread:0:FileTxnLog@204] - Creating new log file: log.109
2016-07-21 04:29:05,165 [myid:] - INFO [ProcessThread(sid:0 cport:11222)::PrepRequestProcessor@647] - Processed session termination for sessionid: 0x10002e296fd0000
2016-07-21 04:29:05,166 [myid:] - INFO [main:ZooKeeper@1313] - Session: 0x10002e296fd0000 closed
2016-07-21 04:29:05,167 [myid:] - INFO [main-EventThread:ClientCnxn$EventThread@513] - EventThread shut down for session: 0x10002e296fd0000
2016-07-21 04:29:05,167 [myid:] - INFO [ConnnectionExpirer:NIOServerCnxnFactory$ConnectionExpirerThread@583] - ConnnectionExpirerThread interrupted
2016-07-21 04:29:05,166 [myid:] - INFO [NIOWorkerThread-7:MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=StandaloneServer_port11222,name1=Connections,name2=127.0.0.1,name3=0x10002e296fd0000]
2016-07-21 04:29:05,167 [myid:] - INFO [NIOServerCxnFactory.SelectorThread-0:NIOServerCnxnFactory$SelectorThread@420] - selector thread exitted run method
2016-07-21 04:29:05,167 [myid:] - INFO [NIOServerCxnFactory.AcceptThread:0.0.0.0/0.0.0.0:11222:NIOServerCnxnFactory$AcceptThread@219] - accept thread exitted run method
2016-07-21 04:29:05,168 [myid:] - INFO [NIOWorkerThread-7:NIOServerCnxn@607] - Closed socket connection for client /127.0.0.1:48682 which had sessionid 0x10002e296fd0000
2016-07-21 04:29:05,168 [myid:] - INFO [main:ZooKeeperServer@498] - shutting down
2016-07-21 04:29:05,168 [myid:] - INFO [main:SessionTrackerImpl@232] - Shutting down
2016-07-21 04:29:05,168 [myid:] - INFO [main:PrepRequestProcessor@965] - Shutting down
2016-07-21 04:29:05,169 [myid:] - INFO [main:SyncRequestProcessor@191] - Shutting down
2016-07-21 04:29:05,169 [myid:] - INFO [ProcessThread(sid:0 cport:11222)::PrepRequestProcessor@154] - PrepRequestProcessor exited loop!
2016-07-21 04:29:05,169 [myid:] - INFO [SyncThread:0:SyncRequestProcessor@169] - SyncRequestProcessor exited!
2016-07-21 04:29:05,169 [myid:] - INFO [main:FinalRequestProcessor@479] - shutdown of request processor complete
2016-07-21 04:29:05,169 [myid:] - INFO [main:MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=StandaloneServer_port11222,name1=InMemoryDataTree]
2016-07-21 04:29:05,170 [myid:] - INFO [main:MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=StandaloneServer_port11222]
2016-07-21 04:29:05,170 [myid:] - INFO [main:FourLetterWordMain@85] - connecting to 127.0.0.1 11222
2016-07-21 04:29:05,172 [myid:] - INFO [main:LoadFromLogTest@115] - Txnlog size: 307248 bytes
2016-07-21 04:29:05,186 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@82] - Memory used 7789
2016-07-21 04:29:05,187 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@87] - Number of threads 11
2016-07-21 04:29:05,187 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@102] - FINISHED TEST METHOD testLoad
2016-07-21 04:29:05,187 [myid:] - INFO [main:ZKTestCase$1@65] - SUCCEEDED testLoad
2016-07-21 04:29:05,187 [myid:] - INFO [main:ZKTestCase$1@60] - FINISHED testLoad
2016-07-21 04:29:05,188 [myid:] - INFO [main:ZKTestCase$1@55] - STARTING testTxnFailure
2016-07-21 04:29:05,188 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@77] - RUNNING TEST METHOD testTxnFailure
2016-07-21 04:29:05,188 [myid:] - INFO [main:LoadFromLogTest@245] - Attempting to create /test/3
2016-07-21 04:29:05,189 [myid:] - INFO [main:LoadFromLogTest@281] - Children: 3 2 1 for /test
2016-07-21 04:29:05,189 [myid:] - INFO [main:LoadFromLogTest@282] - (cverions, pzxid): 3, 3
2016-07-21 04:29:05,189 [myid:] - INFO [main:LoadFromLogTest@319] - Children: 3 2 1 for /test
2016-07-21 04:29:05,189 [myid:] - INFO [main:LoadFromLogTest@320] - (cverions, pzxid): 4, 4
2016-07-21 04:29:05,189 [myid:] - INFO [main:LoadFromLogTest@248] - Attempting to create /test/3
2016-07-21 04:29:05,189 [myid:] - INFO [main:LoadFromLogTest@281] - Children: 3 2 1 for /test
2016-07-21 04:29:05,190 [myid:] - INFO [main:LoadFromLogTest@282] - (cverions, pzxid): 4, 4
2016-07-21 04:29:05,190 [myid:] - INFO [main:LoadFromLogTest@319] - Children: 3 2 1 for /test
2016-07-21 04:29:05,190 [myid:] - INFO [main:LoadFromLogTest@320] - (cverions, pzxid): 5, 5
2016-07-21 04:29:05,190 [myid:] - INFO [main:LoadFromLogTest@252] - Attempting to create /test/3
2016-07-21 04:29:05,190 [myid:] - INFO [main:LoadFromLogTest@281] - Children: 3 2 1 for /test
2016-07-21 04:29:05,190 [myid:] - INFO [main:LoadFromLogTest@282] - (cverions, pzxid): 5, 5
2016-07-21 04:29:05,192 [myid:] - INFO [main:LoadFromLogTest@319] - Children: 3 2 1 for /test
2016-07-21 04:29:05,192 [myid:] - INFO [main:LoadFromLogTest@320] - (cverions, pzxid): 6, 6
2016-07-21 04:29:05,192 [myid:] - INFO [main:LoadFromLogTest@256] - Attempting to create /test/3
2016-07-21 04:29:05,193 [myid:] - INFO [main:LoadFromLogTest@281] - Children: 3 2 1 for /test
2016-07-21 04:29:05,193 [myid:] - INFO [main:LoadFromLogTest@282] - (cverions, pzxid): 6, 6
2016-07-21 04:29:05,193 [myid:] - INFO [main:LoadFromLogTest@319] - Children: 3 2 1 for /test
2016-07-21 04:29:05,193 [myid:] - INFO [main:LoadFromLogTest@320] - (cverions, pzxid): 7, 7
2016-07-21 04:29:05,193 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@82] - Memory used 8013
2016-07-21 04:29:05,193 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@87] - Number of threads 11
2016-07-21 04:29:05,194 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@102] - FINISHED TEST METHOD testTxnFailure
2016-07-21 04:29:05,194 [myid:] - INFO [main:ZKTestCase$1@65] - SUCCEEDED testTxnFailure
2016-07-21 04:29:05,194 [myid:] - INFO [main:ZKTestCase$1@60] - FINISHED testTxnFailure
2016-07-21 04:29:05,194 [myid:] - INFO [main:ZKTestCase$1@55] - STARTING testDatadirAutocreate
2016-07-21 04:29:05,194 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@77] - RUNNING TEST METHOD testDatadirAutocreate
2016-07-21 04:29:05,195 [myid:] - INFO [main:ZooKeeperServer@858] - minSessionTimeout set to 6000
2016-07-21 04:29:05,195 [myid:] - INFO [main:ZooKeeperServer@867] - maxSessionTimeout set to 60000
2016-07-21 04:29:05,195 [myid:] - INFO [main:ZooKeeperServer@156] - Created server with tickTime 3000 minSessionTimeout 6000 maxSessionTimeout 60000 datadir /home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/trunk/build/test/tmp/test6847426618122472335.junit.dir/version-2 snapdir /home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/trunk/build/test/tmp/test6847426618122472335.junit.dir/version-2
2016-07-21 04:29:05,195 [myid:] - INFO [main:NIOServerCnxnFactory@673] - Configuring NIO connection handler with 10s sessionless connection timeout, 1 selector thread(s), 8 worker threads, and 64 kB direct buffers.
2016-07-21 04:29:05,196 [myid:] - INFO [main:NIOServerCnxnFactory@686] - binding to port 0.0.0.0/0.0.0.0:11222
2016-07-21 04:29:05,197 [myid:] - INFO [main:FileTxnSnapLog@298] - Snapshotting: 0x0 to /home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/trunk/build/test/tmp/test6847426618122472335.junit.dir/version-2/snapshot.0
2016-07-21 04:29:05,198 [myid:] - INFO [main:FourLetterWordMain@85] - connecting to 127.0.0.1 11222
2016-07-21 04:29:05,198 [myid:] - INFO [NIOServerCxnFactory.AcceptThread:0.0.0.0/0.0.0.0:11222:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:48685
2016-07-21 04:29:05,199 [myid:] - INFO [NIOWorkerThread-1:NIOServerCnxn@485] - Processing stat command from /127.0.0.1:48685
2016-07-21 04:29:05,199 [myid:] - INFO [NIOWorkerThread-1:StatCommand@49] - Stat command output
2016-07-21 04:29:05,200 [myid:] - INFO [NIOWorkerThread-1:NIOServerCnxn@607] - Closed socket connection for client /127.0.0.1:48685 (no session established for client)
2016-07-21 04:29:05,200 [myid:] - INFO [main:ZooKeeperServer@498] - shutting down
2016-07-21 04:29:05,200 [myid:] - INFO [main:SessionTrackerImpl@232] - Shutting down
2016-07-21 04:29:05,200 [myid:] - INFO [main:PrepRequestProcessor@965] - Shutting down
2016-07-21 04:29:05,200 [myid:] - INFO [main:SyncRequestProcessor@191] - Shutting down
2016-07-21 04:29:05,200 [myid:] - INFO [ProcessThread(sid:0 cport:11222)::PrepRequestProcessor@154] - PrepRequestProcessor exited loop!
2016-07-21 04:29:05,200 [myid:] - INFO [SyncThread:0:SyncRequestProcessor@169] - SyncRequestProcessor exited!
2016-07-21 04:29:05,201 [myid:] - INFO [main:FinalRequestProcessor@479] - shutdown of request processor complete
2016-07-21 04:29:05,201 [myid:] - INFO [main:MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=StandaloneServer_port11222,name1=InMemoryDataTree]
2016-07-21 04:29:05,201 [myid:] - INFO [main:MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=StandaloneServer_port11222]
2016-07-21 04:29:05,202 [myid:] - INFO [ConnnectionExpirer:NIOServerCnxnFactory$ConnectionExpirerThread@583] - ConnnectionExpirerThread interrupted
2016-07-21 04:29:05,202 [myid:] - INFO [NIOServerCxnFactory.SelectorThread-0:NIOServerCnxnFactory$SelectorThread@420] - selector thread exitted run method
2016-07-21 04:29:05,202 [myid:] - INFO [NIOServerCxnFactory.AcceptThread:0.0.0.0/0.0.0.0:11222:NIOServerCnxnFactory$AcceptThread@219] - accept thread exitted run method
2016-07-21 04:29:05,202 [myid:] - INFO [main:FourLetterWordMain@85] - connecting to 127.0.0.1 11222
2016-07-21 04:29:05,203 [myid:] - INFO [main:LoadFromLogTest@517] - Server failed to start - correct behavior org.apache.zookeeper.server.persistence.FileTxnSnapLog$DatadirException: Missing data directory /home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/trunk/build/test/tmp/test6323392488588100984.junit.dir/version-2, automatic data directory creation is disabled (zookeeper.datadir.autocreate is false). Please create this directory manually.
2016-07-21 04:29:05,203 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@82] - Memory used 8605
2016-07-21 04:29:05,203 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@87] - Number of threads 12
2016-07-21 04:29:05,203 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@102] - FINISHED TEST METHOD testDatadirAutocreate
2016-07-21 04:29:05,203 [myid:] - INFO [main:ZKTestCase$1@65] - SUCCEEDED testDatadirAutocreate
2016-07-21 04:29:05,203 [myid:] - INFO [main:ZKTestCase$1@60] - FINISHED testDatadirAutocreate
2016-07-21 04:29:05,204 [myid:] - INFO [main:ZKTestCase$1@55] - STARTING testRestore
2016-07-21 04:29:05,204 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@77] - RUNNING TEST METHOD testRestore
2016-07-21 04:29:05,205 [myid:] - INFO [main:ZooKeeperServer@858] - minSessionTimeout set to 6000
2016-07-21 04:29:05,432 [myid:] - INFO [SessionTracker:SessionTrackerImpl@158] - SessionTrackerImpl exited loop!
2016-07-21 04:29:05,432 [myid:] - INFO [SessionTracker:SessionTrackerImpl@158] - SessionTrackerImpl exited loop!
2016-07-21 04:29:05,432 [myid:] - INFO [SessionTracker:SessionTrackerImpl@158] - SessionTrackerImpl exited loop!
2016-07-21 04:29:05,432 [myid:] - INFO [SessionTracker:SessionTrackerImpl@158] - SessionTrackerImpl exited loop!
2016-07-21 04:29:05,432 [myid:] - INFO [SessionTracker:SessionTrackerImpl@158] - SessionTrackerImpl exited loop!
2016-07-21 04:29:05,432 [myid:] - INFO [SessionTracker:SessionTrackerImpl@158] - SessionTrackerImpl exited loop!
2016-07-21 04:29:13,168 [myid:] - INFO [main:ZooKeeperServer@867] - maxSessionTimeout set to 60000
2016-07-21 04:29:13,169 [myid:] - INFO [main:ZooKeeperServer@156] - Created server with tickTime 3000 minSessionTimeout 6000 maxSessionTimeout 60000 datadir /home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/trunk/build/test/tmp/test4732590081806435620.junit.dir/version-2 snapdir /home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/trunk/build/test/tmp/test4732590081806435620.junit.dir/version-2
2016-07-21 04:29:13,170 [myid:] - INFO [main:NIOServerCnxnFactory@673] - Configuring NIO connection handler with 10s sessionless connection timeout, 1 selector thread(s), 8 worker threads, and 64 kB direct buffers.
2016-07-21 04:29:13,170 [myid:] - INFO [main:NIOServerCnxnFactory@686] - binding to port 0.0.0.0/0.0.0.0:11222
2016-07-21 04:29:13,171 [myid:] - INFO [main:FileTxnSnapLog@298] - Snapshotting: 0x0 to /home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/trunk/build/test/tmp/test4732590081806435620.junit.dir/version-2/snapshot.0
2016-07-21 04:29:13,173 [myid:] - INFO [main:FourLetterWordMain@85] - connecting to 127.0.0.1 11222
2016-07-21 04:29:13,173 [myid:] - INFO [NIOServerCxnFactory.AcceptThread:0.0.0.0/0.0.0.0:11222:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:48687
2016-07-21 04:29:13,175 [myid:] - INFO [NIOWorkerThread-1:NIOServerCnxn@485] - Processing stat command from /127.0.0.1:48687
2016-07-21 04:29:13,175 [myid:] - INFO [NIOWorkerThread-1:StatCommand@49] - Stat command output
2016-07-21 04:29:13,176 [myid:] - INFO [NIOWorkerThread-1:NIOServerCnxn@607] - Closed socket connection for client /127.0.0.1:48687 (no session established for client)
2016-07-21 04:29:13,176 [myid:] - INFO [main:ZooKeeper@855] - Initiating client connection, connectString=127.0.0.1:11222 sessionTimeout=3000 watcher=org.apache.zookeeper.test.LoadFromLogTest@3f9e244f
2016-07-21 04:29:13,177 [myid:127.0.0.1:11222] - INFO [main-SendThread(127.0.0.1:11222):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:11222. Will not attempt to authenticate using SASL (unknown error)
2016-07-21 04:29:13,177 [myid:127.0.0.1:11222] - INFO [main-SendThread(127.0.0.1:11222):ClientCnxn$SendThread@948] - Socket connection established, initiating session, client: /127.0.0.1:48688, server: 127.0.0.1/127.0.0.1:11222
2016-07-21 04:29:13,177 [myid:] - INFO [NIOServerCxnFactory.AcceptThread:0.0.0.0/0.0.0.0:11222:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:48688
2016-07-21 04:29:13,178 [myid:] - INFO [NIOWorkerThread-2:ZooKeeperServer@964] - Client attempting to establish new session at /127.0.0.1:48688
2016-07-21 04:29:13,179 [myid:] - INFO [SyncThread:0:FileTxnLog@204] - Creating new log file: log.1
2016-07-21 04:29:13,182 [myid:] - INFO [SyncThread:0:ZooKeeperServer@678] - Established session 0x10002e2b8440000 with negotiated timeout 6000 for client /127.0.0.1:48688
2016-07-21 04:29:13,182 [myid:127.0.0.1:11222] - INFO [main-SendThread(127.0.0.1:11222):ClientCnxn$SendThread@1381] - Session establishment complete on server 127.0.0.1/127.0.0.1:11222, sessionid = 0x10002e2b8440000, negotiated timeout = 6000
2016-07-21 04:29:13,823 [myid:] - INFO [ProcessThread(sid:0 cport:11222)::PrepRequestProcessor@647] - Processed session termination for sessionid: 0x10002e2b8440000
2016-07-21 04:29:13,824 [myid:] - INFO [main:ZooKeeper@1313] - Session: 0x10002e2b8440000 closed
2016-07-21 04:29:13,824 [myid:] - INFO [main-EventThread:ClientCnxn$EventThread@513] - EventThread shut down for session: 0x10002e2b8440000
2016-07-21 04:29:13,824 [myid:] - INFO [NIOWorkerThread-7:MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=StandaloneServer_port11222,name1=Connections,name2=127.0.0.1,name3=0x10002e2b8440000]
2016-07-21 04:29:13,824 [myid:] - INFO [main:LoadFromLogTest@387] - Set lastProcessedZxid to 293
2016-07-21 04:29:13,825 [myid:] - INFO [NIOWorkerThread-7:NIOServerCnxn@607] - Closed socket connection for client /127.0.0.1:48688 which had sessionid 0x10002e2b8440000
2016-07-21 04:29:13,825 [myid:] - INFO [main:FileTxnSnapLog@298] - Snapshotting: 0x125 to /home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/trunk/build/test/tmp/test4732590081806435620.junit.dir/version-2/snapshot.125
2016-07-21 04:29:13,830 [myid:] - INFO [main:ZooKeeperServer@498] - shutting down
2016-07-21 04:29:13,830 [myid:] - INFO [main:SessionTrackerImpl@232] - Shutting down
2016-07-21 04:29:13,830 [myid:] - INFO [main:PrepRequestProcessor@965] - Shutting down
2016-07-21 04:29:13,830 [myid:] - INFO [main:SyncRequestProcessor@191] - Shutting down
2016-07-21 04:29:13,830 [myid:] - INFO [ProcessThread(sid:0 cport:11222)::PrepRequestProcessor@154] - PrepRequestProcessor exited loop!
2016-07-21 04:29:13,830 [myid:] - INFO [SyncThread:0:SyncRequestProcessor@169] - SyncRequestProcessor exited!
2016-07-21 04:29:13,831 [myid:] - INFO [main:FinalRequestProcessor@479] - shutdown of request processor complete
2016-07-21 04:29:13,831 [myid:] - INFO [main:MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=StandaloneServer_port11222,name1=InMemoryDataTree]
2016-07-21 04:29:13,831 [myid:] - INFO [main:MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=StandaloneServer_port11222]
2016-07-21 04:29:13,832 [myid:] - INFO [ConnnectionExpirer:NIOServerCnxnFactory$ConnectionExpirerThread@583] - ConnnectionExpirerThread interrupted
2016-07-21 04:29:13,834 [myid:] - INFO [NIOServerCxnFactory.SelectorThread-0:NIOServerCnxnFactory$SelectorThread@420] - selector thread exitted run method
2016-07-21 04:29:13,834 [myid:] - INFO [NIOServerCxnFactory.AcceptThread:0.0.0.0/0.0.0.0:11222:NIOServerCnxnFactory$AcceptThread@219] - accept thread exitted run method
2016-07-21 04:29:13,835 [myid:] - INFO [main:ZooKeeperServer@858] - minSessionTimeout set to 6000
2016-07-21 04:29:13,835 [myid:] - INFO [main:ZooKeeperServer@867] - maxSessionTimeout set to 60000
2016-07-21 04:29:13,835 [myid:] - INFO [main:ZooKeeperServer@156] - Created server with tickTime 3000 minSessionTimeout 6000 maxSessionTimeout 60000 datadir /home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/trunk/build/test/tmp/test4732590081806435620.junit.dir/version-2 snapdir /home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/trunk/build/test/tmp/test4732590081806435620.junit.dir/version-2
2016-07-21 04:29:13,836 [myid:] - INFO [main:NIOServerCnxnFactory@673] - Configuring NIO connection handler with 10s sessionless connection timeout, 1 selector thread(s), 8 worker threads, and 64 kB direct buffers.
2016-07-21 04:29:13,836 [myid:] - INFO [main:NIOServerCnxnFactory@686] - binding to port 0.0.0.0/0.0.0.0:11222
2016-07-21 04:29:13,837 [myid:] - INFO [main:FileSnap@83] - Reading snapshot /home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/trunk/build/test/tmp/test4732590081806435620.junit.dir/version-2/snapshot.125
2016-07-21 04:29:13,858 [myid:] - INFO [main:FileTxnSnapLog@298] - Snapshotting: 0x12f to /home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/trunk/build/test/tmp/test4732590081806435620.junit.dir/version-2/snapshot.12f
2016-07-21 04:29:13,862 [myid:] - INFO [main:FourLetterWordMain@85] - connecting to 127.0.0.1 11222
2016-07-21 04:29:14,432 [myid:] - INFO [SessionTracker:SessionTrackerImpl@158] - SessionTrackerImpl exited loop!
2016-07-21 04:29:19,360 [myid:] - INFO [NIOServerCxnFactory.AcceptThread:0.0.0.0/0.0.0.0:11222:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:48689
2016-07-21 04:29:19,362 [myid:] - INFO [NIOWorkerThread-1:NIOServerCnxn@485] - Processing stat command from /127.0.0.1:48689
2016-07-21 04:29:19,362 [myid:] - INFO [NIOWorkerThread-1:StatCommand@49] - Stat command output
2016-07-21 04:29:19,362 [myid:] - INFO [NIOWorkerThread-1:NIOServerCnxn@607] - Closed socket connection for client /127.0.0.1:48689 (no session established for client)
2016-07-21 04:29:19,363 [myid:] - INFO [main:ZooKeeper@855] - Initiating client connection, connectString=127.0.0.1:11222 sessionTimeout=3000 watcher=org.apache.zookeeper.test.LoadFromLogTest@3f9e244f
2016-07-21 04:29:19,364 [myid:127.0.0.1:11222] - INFO [main-SendThread(127.0.0.1:11222):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:11222. Will not attempt to authenticate using SASL (unknown error)
2016-07-21 04:29:19,364 [myid:] - INFO [NIOServerCxnFactory.AcceptThread:0.0.0.0/0.0.0.0:11222:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:48690
2016-07-21 04:29:19,364 [myid:127.0.0.1:11222] - INFO [main-SendThread(127.0.0.1:11222):ClientCnxn$SendThread@948] - Socket connection established, initiating session, client: /127.0.0.1:48690, server: 127.0.0.1/127.0.0.1:11222
2016-07-21 04:29:19,365 [myid:] - INFO [NIOWorkerThread-2:ZooKeeperServer@964] - Client attempting to establish new session at /127.0.0.1:48690
2016-07-21 04:29:19,366 [myid:] - INFO [SyncThread:0:FileTxnLog@204] - Creating new log file: log.130
2016-07-21 04:29:19,368 [myid:] - INFO [SyncThread:0:ZooKeeperServer@678] - Established session 0x10002e2baf60000 with negotiated timeout 6000 for client /127.0.0.1:48690
2016-07-21 04:29:19,369 [myid:127.0.0.1:11222] - INFO [main-SendThread(127.0.0.1:11222):ClientCnxn$SendThread@1381] - Session establishment complete on server 127.0.0.1/127.0.0.1:11222, sessionid = 0x10002e2baf60000, negotiated timeout = 6000
2016-07-21 04:29:19,574 [myid:] - INFO [ProcessThread(sid:0 cport:11222)::PrepRequestProcessor@647] - Processed session termination for sessionid: 0x10002e2baf60000
2016-07-21 04:29:19,575 [myid:] - INFO [main:ZooKeeper@1313] - Session: 0x10002e2baf60000 closed
2016-07-21 04:29:19,575 [myid:] - INFO [main-EventThread:ClientCnxn$EventThread@513] - EventThread shut down for session: 0x10002e2baf60000
2016-07-21 04:29:19,575 [myid:] - INFO [NIOWorkerThread-1:MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=StandaloneServer_port11222,name1=Connections,name2=127.0.0.1,name3=0x10002e2baf60000]
2016-07-21 04:29:19,575 [myid:] - INFO [main:LoadFromLogTest@420] - Expected /invalidsnap/test-0000000300 found /invalidsnap/test-0000000300
2016-07-21 04:29:19,576 [myid:] - INFO [NIOWorkerThread-1:NIOServerCnxn@607] - Closed socket connection for client /127.0.0.1:48690 which had sessionid 0x10002e2baf60000
2016-07-21 04:29:19,576 [myid:] - INFO [ConnnectionExpirer:NIOServerCnxnFactory$ConnectionExpirerThread@583] - ConnnectionExpirerThread interrupted
2016-07-21 04:29:19,578 [myid:] - INFO [NIOServerCxnFactory.SelectorThread-0:NIOServerCnxnFactory$SelectorThread@420] - selector thread exitted run method
2016-07-21 04:29:19,578 [myid:] - INFO [NIOServerCxnFactory.AcceptThread:0.0.0.0/0.0.0.0:11222:NIOServerCnxnFactory$AcceptThread@219] - accept thread exitted run method
2016-07-21 04:29:19,578 [myid:] - INFO [main:ZooKeeperServer@498] - shutting down
2016-07-21 04:29:19,578 [myid:] - INFO [main:SessionTrackerImpl@232] - Shutting down
2016-07-21 04:29:19,578 [myid:] - INFO [main:PrepRequestProcessor@965] - Shutting down
2016-07-21 04:29:19,578 [myid:] - INFO [main:SyncRequestProcessor@191] - Shutting down
2016-07-21 04:29:19,578 [myid:] - INFO [ProcessThread(sid:0 cport:11222)::PrepRequestProcessor@154] - PrepRequestProcessor exited loop!
2016-07-21 04:29:19,579 [myid:] - INFO [SyncThread:0:SyncRequestProcessor@169] - SyncRequestProcessor exited!
2016-07-21 04:29:19,579 [myid:] - INFO [main:FinalRequestProcessor@479] - shutdown of request processor complete
2016-07-21 04:29:19,579 [myid:] - INFO [main:MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=StandaloneServer_port11222,name1=InMemoryDataTree]
2016-07-21 04:29:19,580 [myid:] - INFO [main:MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=StandaloneServer_port11222]
2016-07-21 04:29:19,580 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@82] - Memory used 16954
2016-07-21 04:29:19,580 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@87] - Number of threads 6
2016-07-21 04:29:19,580 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@102] - FINISHED TEST METHOD testRestore
2016-07-21 04:29:19,580 [myid:] - INFO [main:ZKTestCase$1@65] - SUCCEEDED testRestore
2016-07-21 04:29:19,580 [myid:] - INFO [main:ZKTestCase$1@60] - FINISHED testRestore
2016-07-21 04:29:19,581 [myid:] - INFO [main:ZKTestCase$1@55] - STARTING testLoadFailure
2016-07-21 04:29:19,581 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@77] - RUNNING TEST METHOD testLoadFailure
2016-07-21 04:29:19,581 [myid:] - INFO [main:ZooKeeperServer@858] - minSessionTimeout set to 6000
2016-07-21 04:29:19,581 [myid:] - INFO [main:ZooKeeperServer@867] - maxSessionTimeout set to 60000
2016-07-21 04:29:19,582 [myid:] - INFO [main:ZooKeeperServer@156] - Created server with tickTime 3000 minSessionTimeout 6000 maxSessionTimeout 60000 datadir /home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/trunk/build/test/tmp/test436661564740529918.junit.dir/version-2 snapdir /home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/trunk/build/test/tmp/test436661564740529918.junit.dir/version-2
2016-07-21 04:29:19,582 [myid:] - INFO [main:NIOServerCnxnFactory@673] - Configuring NIO connection handler with 10s sessionless connection timeout, 1 selector thread(s), 8 worker threads, and 64 kB direct buffers.
2016-07-21 04:29:19,582 [myid:] - INFO [main:NIOServerCnxnFactory@686] - binding to port 0.0.0.0/0.0.0.0:11222
2016-07-21 04:29:19,583 [myid:] - INFO [main:FileTxnSnapLog@298] - Snapshotting: 0x0 to /home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/trunk/build/test/tmp/test436661564740529918.junit.dir/version-2/snapshot.0
2016-07-21 04:29:19,584 [myid:] - INFO [main:FourLetterWordMain@85] - connecting to 127.0.0.1 11222
2016-07-21 04:29:19,585 [myid:] - INFO [NIOServerCxnFactory.AcceptThread:0.0.0.0/0.0.0.0:11222:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:48691
2016-07-21 04:29:19,586 [myid:] - INFO [NIOWorkerThread-1:NIOServerCnxn@485] - Processing stat command from /127.0.0.1:48691
2016-07-21 04:29:19,586 [myid:] - INFO [NIOWorkerThread-1:StatCommand@49] - Stat command output
2016-07-21 04:29:19,586 [myid:] - INFO [NIOWorkerThread-1:NIOServerCnxn@607] - Closed socket connection for client /127.0.0.1:48691 (no session established for client)
2016-07-21 04:29:19,586 [myid:] - INFO [main:ZooKeeper@855] - Initiating client connection, connectString=127.0.0.1:11222 sessionTimeout=3000 watcher=org.apache.zookeeper.test.LoadFromLogTest@1b2090ab
2016-07-21 04:29:19,587 [myid:127.0.0.1:11222] - INFO [main-SendThread(127.0.0.1:11222):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:11222. Will not attempt to authenticate using SASL (unknown error)
2016-07-21 04:29:19,588 [myid:] - INFO [NIOServerCxnFactory.AcceptThread:0.0.0.0/0.0.0.0:11222:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:48692
2016-07-21 04:29:19,588 [myid:127.0.0.1:11222] - INFO [main-SendThread(127.0.0.1:11222):ClientCnxn$SendThread@948] - Socket connection established, initiating session, client: /127.0.0.1:48692, server: 127.0.0.1/127.0.0.1:11222
2016-07-21 04:29:20,432 [myid:] - INFO [SessionTracker:SessionTrackerImpl@158] - SessionTrackerImpl exited loop!
2016-07-21 04:29:27,620 [myid:127.0.0.1:11222] - WARN [main-SendThread(127.0.0.1:11222):ClientCnxn$SendThread@1181] - Client session timed out, have not heard from server in 8032ms for sessionid 0x0
2016-07-21 04:29:27,621 [myid:127.0.0.1:11222] - INFO [main-SendThread(127.0.0.1:11222):ClientCnxn$SendThread@1229] - Client session timed out, have not heard from server in 8032ms for sessionid 0x0, closing socket connection and attempting reconnect
2016-07-21 04:29:27,621 [myid:] - INFO [NIOWorkerThread-2:ZooKeeperServer@964] - Client attempting to establish new session at /127.0.0.1:48692
2016-07-21 04:29:27,622 [myid:] - INFO [SyncThread:0:FileTxnLog@204] - Creating new log file: log.1
2016-07-21 04:29:27,624 [myid:] - INFO [SyncThread:0:ZooKeeperServer@678] - Established session 0x10002e2d1500000 with negotiated timeout 6000 for client /127.0.0.1:48692
2016-07-21 04:29:27,625 [myid:] - WARN [NIOWorkerThread-4:NIOServerCnxn@365] - Unable to read additional data from client sessionid 0x10002e2d1500000, likely client has closed socket
2016-07-21 04:29:27,626 [myid:] - INFO [NIOWorkerThread-4:MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=StandaloneServer_port11222,name1=Connections,name2=127.0.0.1,name3=0x10002e2d1500000]
2016-07-21 04:29:27,626 [myid:] - INFO [NIOWorkerThread-4:NIOServerCnxn@607] - Closed socket connection for client /127.0.0.1:48692 which had sessionid 0x10002e2d1500000
2016-07-21 04:29:28,739 [myid:127.0.0.1:11222] - INFO [main-SendThread(127.0.0.1:11222):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:11222. Will not attempt to authenticate using SASL (unknown error)
2016-07-21 04:29:28,740 [myid:] - INFO [NIOServerCxnFactory.AcceptThread:0.0.0.0/0.0.0.0:11222:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:48693
2016-07-21 04:29:28,740 [myid:127.0.0.1:11222] - INFO [main-SendThread(127.0.0.1:11222):ClientCnxn$SendThread@948] - Socket connection established, initiating session, client: /127.0.0.1:48693, server: 127.0.0.1/127.0.0.1:11222
2016-07-21 04:29:28,741 [myid:] - INFO [NIOWorkerThread-5:ZooKeeperServer@964] - Client attempting to establish new session at /127.0.0.1:48693
2016-07-21 04:29:28,742 [myid:] - INFO [SyncThread:0:ZooKeeperServer@678] - Established session 0x10002e2d1500001 with negotiated timeout 6000 for client /127.0.0.1:48693
2016-07-21 04:29:28,743 [myid:127.0.0.1:11222] - INFO [main-SendThread(127.0.0.1:11222):ClientCnxn$SendThread@1381] - Session establishment complete on server 127.0.0.1/127.0.0.1:11222, sessionid = 0x10002e2d1500001, negotiated timeout = 6000
2016-07-21 04:29:28,744 [myid:] - INFO [ProcessThread(sid:0 cport:11222)::PrepRequestProcessor@647] - Processed session termination for sessionid: 0x10002e2d1500001
2016-07-21 04:29:28,746 [myid:] - INFO [NIOWorkerThread-8:MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=StandaloneServer_port11222,name1=Connections,name2=127.0.0.1,name3=0x10002e2d1500001]
2016-07-21 04:29:28,746 [myid:] - INFO [main-EventThread:ClientCnxn$EventThread@513] - EventThread shut down for session: 0x10002e2d1500001
2016-07-21 04:29:28,746 [myid:] - INFO [NIOWorkerThread-8:NIOServerCnxn@607] - Closed socket connection for client /127.0.0.1:48693 which had sessionid 0x10002e2d1500001
2016-07-21 04:29:28,746 [myid:] - INFO [main:ZooKeeper@1313] - Session: 0x10002e2d1500001 closed
2016-07-21 04:29:28,748 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@98] - TEST METHOD FAILED testLoadFailure
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /data-
at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:1412)
at org.apache.zookeeper.test.LoadFromLogTest.testLoadFailure(LoadFromLogTest.java:157)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:79)
at org.junit.rules.TestWatchman$1.evaluate(TestWatchman.java:53)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:38)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:518)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1052)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:906)
2016-07-21 04:29:28,750 [myid:] - INFO [main:ZKTestCase$1@70] - FAILED testLoadFailure
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /data-
at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:1412)
at org.apache.zookeeper.test.LoadFromLogTest.testLoadFailure(LoadFromLogTest.java:157)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:79)
at org.junit.rules.TestWatchman$1.evaluate(TestWatchman.java:53)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:38)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:518)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1052)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:906)
2016-07-21 04:29:29,830 [myid:] - INFO [main:ZKTestCase$1@60] - FINISHED testLoadFailure
{noformat}
flaky, flaky-test 9223372036854775807 No Perforce job exists for this issue. 2 9223372036854775807
3 years, 28 weeks ago
Reviewed
0|i31cjz:
ZooKeeper ZOOKEEPER-2483

Flaky Test: org.apache.zookeeper.test.LETest.testLE

Test Closed Major Duplicate Michael Han Michael Han Michael Han 21/Jul/16 17:45   17/May/17 23:44 27/Jul/16 17:40 3.4.8, 3.5.2 3.5.3 server, tests   0 2   From https://builds.apache.org/job/ZooKeeper_branch34/1587/

{noformat}
Error Message

Threads didn't join
Stacktrace

junit.framework.AssertionFailedError: Threads didn't join
at org.apache.zookeeper.test.LETest.testLE(LETest.java:120)
at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:55)
Standard Output

2016-07-21 06:02:29,408 [myid:] - INFO [main:ZKTestCase$1@50] - STARTING testLE
2016-07-21 06:02:29,413 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@53] - RUNNING TEST METHOD testLE
2016-07-21 06:02:29,417 [myid:] - INFO [main:PortAssignment@32] - assigning port 11221
2016-07-21 06:02:29,434 [myid:] - INFO [main:QuorumPeer$QuorumServer@149] - Resolved hostname: 127.0.0.1 to address: /127.0.0.1
2016-07-21 06:02:29,450 [myid:] - INFO [main:PortAssignment@32] - assigning port 11222
2016-07-21 06:02:29,450 [myid:] - INFO [main:PortAssignment@32] - assigning port 11223
2016-07-21 06:02:29,450 [myid:] - INFO [main:QuorumPeer$QuorumServer@149] - Resolved hostname: 127.0.0.1 to address: /127.0.0.1
2016-07-21 06:02:29,450 [myid:] - INFO [main:PortAssignment@32] - assigning port 11224
2016-07-21 06:02:29,451 [myid:] - INFO [main:PortAssignment@32] - assigning port 11225
2016-07-21 06:02:29,451 [myid:] - INFO [main:QuorumPeer$QuorumServer@149] - Resolved hostname: 127.0.0.1 to address: /127.0.0.1
2016-07-21 06:02:29,451 [myid:] - INFO [main:PortAssignment@32] - assigning port 11226
2016-07-21 06:02:29,451 [myid:] - INFO [main:PortAssignment@32] - assigning port 11227
2016-07-21 06:02:29,452 [myid:] - INFO [main:QuorumPeer$QuorumServer@149] - Resolved hostname: 127.0.0.1 to address: /127.0.0.1
2016-07-21 06:02:29,452 [myid:] - INFO [main:PortAssignment@32] - assigning port 11228
2016-07-21 06:02:29,452 [myid:] - INFO [main:PortAssignment@32] - assigning port 11229
2016-07-21 06:02:29,452 [myid:] - INFO [main:QuorumPeer$QuorumServer@149] - Resolved hostname: 127.0.0.1 to address: /127.0.0.1
2016-07-21 06:02:29,455 [myid:] - INFO [main:PortAssignment@32] - assigning port 11230
2016-07-21 06:02:29,455 [myid:] - INFO [main:PortAssignment@32] - assigning port 11231
2016-07-21 06:02:29,456 [myid:] - INFO [main:QuorumPeer$QuorumServer@149] - Resolved hostname: 127.0.0.1 to address: /127.0.0.1
2016-07-21 06:02:29,456 [myid:] - INFO [main:PortAssignment@32] - assigning port 11232
2016-07-21 06:02:29,456 [myid:] - INFO [main:PortAssignment@32] - assigning port 11233
2016-07-21 06:02:29,456 [myid:] - INFO [main:QuorumPeer$QuorumServer@149] - Resolved hostname: 127.0.0.1 to address: /127.0.0.1
2016-07-21 06:02:29,457 [myid:] - INFO [main:PortAssignment@32] - assigning port 11234
2016-07-21 06:02:29,457 [myid:] - INFO [main:PortAssignment@32] - assigning port 11235
2016-07-21 06:02:29,457 [myid:] - INFO [main:QuorumPeer$QuorumServer@149] - Resolved hostname: 127.0.0.1 to address: /127.0.0.1
2016-07-21 06:02:29,458 [myid:] - INFO [main:PortAssignment@32] - assigning port 11236
2016-07-21 06:02:29,458 [myid:] - INFO [main:PortAssignment@32] - assigning port 11237
2016-07-21 06:02:29,458 [myid:] - INFO [main:QuorumPeer$QuorumServer@149] - Resolved hostname: 127.0.0.1 to address: /127.0.0.1
2016-07-21 06:02:29,458 [myid:] - INFO [main:PortAssignment@32] - assigning port 11238
2016-07-21 06:02:29,459 [myid:] - INFO [main:PortAssignment@32] - assigning port 11239
2016-07-21 06:02:29,459 [myid:] - INFO [main:QuorumPeer$QuorumServer@149] - Resolved hostname: 127.0.0.1 to address: /127.0.0.1
2016-07-21 06:02:29,459 [myid:] - INFO [main:PortAssignment@32] - assigning port 11240
2016-07-21 06:02:29,459 [myid:] - INFO [main:PortAssignment@32] - assigning port 11241
2016-07-21 06:02:29,460 [myid:] - INFO [main:QuorumPeer$QuorumServer@149] - Resolved hostname: 127.0.0.1 to address: /127.0.0.1
2016-07-21 06:02:29,460 [myid:] - INFO [main:PortAssignment@32] - assigning port 11242
2016-07-21 06:02:29,460 [myid:] - INFO [main:PortAssignment@32] - assigning port 11243
2016-07-21 06:02:29,460 [myid:] - INFO [main:QuorumPeer$QuorumServer@149] - Resolved hostname: 127.0.0.1 to address: /127.0.0.1
2016-07-21 06:02:29,461 [myid:] - INFO [main:PortAssignment@32] - assigning port 11244
2016-07-21 06:02:29,461 [myid:] - INFO [main:PortAssignment@32] - assigning port 11245
2016-07-21 06:02:29,461 [myid:] - INFO [main:QuorumPeer$QuorumServer@149] - Resolved hostname: 127.0.0.1 to address: /127.0.0.1
2016-07-21 06:02:29,461 [myid:] - INFO [main:PortAssignment@32] - assigning port 11246
2016-07-21 06:02:29,462 [myid:] - INFO [main:PortAssignment@32] - assigning port 11247
2016-07-21 06:02:29,462 [myid:] - INFO [main:QuorumPeer$QuorumServer@149] - Resolved hostname: 127.0.0.1 to address: /127.0.0.1
2016-07-21 06:02:29,462 [myid:] - INFO [main:PortAssignment@32] - assigning port 11248
2016-07-21 06:02:29,462 [myid:] - INFO [main:PortAssignment@32] - assigning port 11249
2016-07-21 06:02:29,462 [myid:] - INFO [main:QuorumPeer$QuorumServer@149] - Resolved hostname: 127.0.0.1 to address: /127.0.0.1
2016-07-21 06:02:29,463 [myid:] - INFO [main:PortAssignment@32] - assigning port 11250
2016-07-21 06:02:29,463 [myid:] - INFO [main:PortAssignment@32] - assigning port 11251
2016-07-21 06:02:29,463 [myid:] - INFO [main:QuorumPeer$QuorumServer@149] - Resolved hostname: 127.0.0.1 to address: /127.0.0.1
2016-07-21 06:02:29,464 [myid:] - INFO [main:PortAssignment@32] - assigning port 11252
2016-07-21 06:02:29,464 [myid:] - INFO [main:PortAssignment@32] - assigning port 11253
2016-07-21 06:02:29,464 [myid:] - INFO [main:QuorumPeer$QuorumServer@149] - Resolved hostname: 127.0.0.1 to address: /127.0.0.1
2016-07-21 06:02:29,464 [myid:] - INFO [main:PortAssignment@32] - assigning port 11254
2016-07-21 06:02:29,464 [myid:] - INFO [main:PortAssignment@32] - assigning port 11255
2016-07-21 06:02:29,465 [myid:] - INFO [main:QuorumPeer$QuorumServer@149] - Resolved hostname: 127.0.0.1 to address: /127.0.0.1
2016-07-21 06:02:29,465 [myid:] - INFO [main:PortAssignment@32] - assigning port 11256
2016-07-21 06:02:29,465 [myid:] - INFO [main:PortAssignment@32] - assigning port 11257
2016-07-21 06:02:29,465 [myid:] - INFO [main:QuorumPeer$QuorumServer@149] - Resolved hostname: 127.0.0.1 to address: /127.0.0.1
2016-07-21 06:02:29,466 [myid:] - INFO [main:PortAssignment@32] - assigning port 11258
2016-07-21 06:02:29,466 [myid:] - INFO [main:PortAssignment@32] - assigning port 11259
2016-07-21 06:02:29,466 [myid:] - INFO [main:QuorumPeer$QuorumServer@149] - Resolved hostname: 127.0.0.1 to address: /127.0.0.1
2016-07-21 06:02:29,466 [myid:] - INFO [main:PortAssignment@32] - assigning port 11260
2016-07-21 06:02:29,467 [myid:] - INFO [main:PortAssignment@32] - assigning port 11261
2016-07-21 06:02:29,467 [myid:] - INFO [main:QuorumPeer$QuorumServer@149] - Resolved hostname: 127.0.0.1 to address: /127.0.0.1
2016-07-21 06:02:29,468 [myid:] - INFO [main:PortAssignment@32] - assigning port 11262
2016-07-21 06:02:29,468 [myid:] - INFO [main:PortAssignment@32] - assigning port 11263
2016-07-21 06:02:29,468 [myid:] - INFO [main:QuorumPeer$QuorumServer@149] - Resolved hostname: 127.0.0.1 to address: /127.0.0.1
2016-07-21 06:02:29,469 [myid:] - INFO [main:PortAssignment@32] - assigning port 11264
2016-07-21 06:02:29,469 [myid:] - INFO [main:PortAssignment@32] - assigning port 11265
2016-07-21 06:02:29,469 [myid:] - INFO [main:QuorumPeer$QuorumServer@149] - Resolved hostname: 127.0.0.1 to address: /127.0.0.1
2016-07-21 06:02:29,470 [myid:] - INFO [main:PortAssignment@32] - assigning port 11266
2016-07-21 06:02:29,470 [myid:] - INFO [main:PortAssignment@32] - assigning port 11267
2016-07-21 06:02:29,470 [myid:] - INFO [main:QuorumPeer$QuorumServer@149] - Resolved hostname: 127.0.0.1 to address: /127.0.0.1
2016-07-21 06:02:29,471 [myid:] - INFO [main:PortAssignment@32] - assigning port 11268
2016-07-21 06:02:29,471 [myid:] - INFO [main:PortAssignment@32] - assigning port 11269
2016-07-21 06:02:29,471 [myid:] - INFO [main:QuorumPeer$QuorumServer@149] - Resolved hostname: 127.0.0.1 to address: /127.0.0.1
2016-07-21 06:02:29,472 [myid:] - INFO [main:PortAssignment@32] - assigning port 11270
2016-07-21 06:02:29,472 [myid:] - INFO [main:PortAssignment@32] - assigning port 11271
2016-07-21 06:02:29,472 [myid:] - INFO [main:QuorumPeer$QuorumServer@149] - Resolved hostname: 127.0.0.1 to address: /127.0.0.1
2016-07-21 06:02:29,473 [myid:] - INFO [main:PortAssignment@32] - assigning port 11272
2016-07-21 06:02:29,473 [myid:] - INFO [main:PortAssignment@32] - assigning port 11273
2016-07-21 06:02:29,473 [myid:] - INFO [main:QuorumPeer$QuorumServer@149] - Resolved hostname: 127.0.0.1 to address: /127.0.0.1
2016-07-21 06:02:29,474 [myid:] - INFO [main:PortAssignment@32] - assigning port 11274
2016-07-21 06:02:29,474 [myid:] - INFO [main:PortAssignment@32] - assigning port 11275
2016-07-21 06:02:29,474 [myid:] - INFO [main:QuorumPeer$QuorumServer@149] - Resolved hostname: 127.0.0.1 to address: /127.0.0.1
2016-07-21 06:02:29,474 [myid:] - INFO [main:PortAssignment@32] - assigning port 11276
2016-07-21 06:02:29,475 [myid:] - INFO [main:PortAssignment@32] - assigning port 11277
2016-07-21 06:02:29,475 [myid:] - INFO [main:QuorumPeer$QuorumServer@149] - Resolved hostname: 127.0.0.1 to address: /127.0.0.1
2016-07-21 06:02:29,475 [myid:] - INFO [main:PortAssignment@32] - assigning port 11278
2016-07-21 06:02:29,475 [myid:] - INFO [main:PortAssignment@32] - assigning port 11279
2016-07-21 06:02:29,476 [myid:] - INFO [main:QuorumPeer$QuorumServer@149] - Resolved hostname: 127.0.0.1 to address: /127.0.0.1
2016-07-21 06:02:29,476 [myid:] - INFO [main:PortAssignment@32] - assigning port 11280
2016-07-21 06:02:29,505 [myid:] - INFO [main:NIOServerCnxnFactory@89] - binding to port 0.0.0.0/0.0.0.0:11222
2016-07-21 06:02:29,530 [myid:] - INFO [main:QuorumPeer@533] - currentEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
2016-07-21 06:02:29,534 [myid:] - INFO [main:QuorumPeer@548] - acceptedEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
2016-07-21 06:02:29,543 [myid:] - INFO [main:NIOServerCnxnFactory@89] - binding to port 0.0.0.0/0.0.0.0:11224
2016-07-21 06:02:29,545 [myid:] - INFO [main:QuorumPeer@533] - currentEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
2016-07-21 06:02:29,546 [myid:] - INFO [main:QuorumPeer@548] - acceptedEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
2016-07-21 06:02:29,549 [myid:] - INFO [main:NIOServerCnxnFactory@89] - binding to port 0.0.0.0/0.0.0.0:11226
2016-07-21 06:02:29,550 [myid:] - INFO [main:QuorumPeer@533] - currentEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
2016-07-21 06:02:29,551 [myid:] - INFO [main:QuorumPeer@548] - acceptedEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
2016-07-21 06:02:29,553 [myid:] - INFO [main:NIOServerCnxnFactory@89] - binding to port 0.0.0.0/0.0.0.0:11228
2016-07-21 06:02:29,554 [myid:] - INFO [main:QuorumPeer@533] - currentEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
2016-07-21 06:02:29,555 [myid:] - INFO [main:QuorumPeer@548] - acceptedEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
2016-07-21 06:02:29,558 [myid:] - INFO [main:NIOServerCnxnFactory@89] - binding to port 0.0.0.0/0.0.0.0:11230
2016-07-21 06:02:29,558 [myid:] - INFO [main:QuorumPeer@533] - currentEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
2016-07-21 06:02:29,559 [myid:] - INFO [main:QuorumPeer@548] - acceptedEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
2016-07-21 06:02:29,562 [myid:] - INFO [main:NIOServerCnxnFactory@89] - binding to port 0.0.0.0/0.0.0.0:11232
2016-07-21 06:02:29,563 [myid:] - INFO [main:QuorumPeer@533] - currentEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
2016-07-21 06:02:29,564 [myid:] - INFO [main:QuorumPeer@548] - acceptedEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
2016-07-21 06:02:29,566 [myid:] - INFO [main:NIOServerCnxnFactory@89] - binding to port 0.0.0.0/0.0.0.0:11234
2016-07-21 06:02:29,567 [myid:] - INFO [main:QuorumPeer@533] - currentEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
2016-07-21 06:02:29,568 [myid:] - INFO [main:QuorumPeer@548] - acceptedEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
2016-07-21 06:02:29,571 [myid:] - INFO [main:NIOServerCnxnFactory@89] - binding to port 0.0.0.0/0.0.0.0:11236
2016-07-21 06:02:29,572 [myid:] - INFO [main:QuorumPeer@533] - currentEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
2016-07-21 06:02:29,573 [myid:] - INFO [main:QuorumPeer@548] - acceptedEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
2016-07-21 06:02:29,575 [myid:] - INFO [main:NIOServerCnxnFactory@89] - binding to port 0.0.0.0/0.0.0.0:11238
2016-07-21 06:02:29,576 [myid:] - INFO [main:QuorumPeer@533] - currentEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
2016-07-21 06:02:29,577 [myid:] - INFO [main:QuorumPeer@548] - acceptedEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
2016-07-21 06:02:29,580 [myid:] - INFO [main:NIOServerCnxnFactory@89] - binding to port 0.0.0.0/0.0.0.0:11240
2016-07-21 06:02:29,581 [myid:] - INFO [main:QuorumPeer@533] - currentEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
2016-07-21 06:02:29,582 [myid:] - INFO [main:QuorumPeer@548] - acceptedEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
2016-07-21 06:02:29,585 [myid:] - INFO [main:NIOServerCnxnFactory@89] - binding to port 0.0.0.0/0.0.0.0:11242
2016-07-21 06:02:29,586 [myid:] - INFO [main:QuorumPeer@533] - currentEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
2016-07-21 06:02:29,587 [myid:] - INFO [main:QuorumPeer@548] - acceptedEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
2016-07-21 06:02:29,589 [myid:] - INFO [main:NIOServerCnxnFactory@89] - binding to port 0.0.0.0/0.0.0.0:11244
2016-07-21 06:02:29,590 [myid:] - INFO [main:QuorumPeer@533] - currentEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
2016-07-21 06:02:29,592 [myid:] - INFO [main:QuorumPeer@548] - acceptedEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
2016-07-21 06:02:29,594 [myid:] - INFO [main:NIOServerCnxnFactory@89] - binding to port 0.0.0.0/0.0.0.0:11246
2016-07-21 06:02:29,595 [myid:] - INFO [Thread-0:LeaderElection@187] - Server address: /127.0.0.1:11221
2016-07-21 06:02:29,597 [myid:] - WARN [Thread-11:MBeanRegistry@104] - Failed to register MBean LeaderElection
2016-07-21 06:02:29,597 [myid:] - WARN [Thread-9:MBeanRegistry@104] - Failed to register MBean LeaderElection
2016-07-21 06:02:29,595 [myid:] - WARN [Thread-12:MBeanRegistry@104] - Failed to register MBean LeaderElection
2016-07-21 06:02:29,596 [myid:] - WARN [Thread-10:MBeanRegistry@104] - Failed to register MBean LeaderElection
2016-07-21 06:02:29,597 [myid:] - INFO [main:QuorumPeer@533] - currentEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
2016-07-21 06:02:29,598 [myid:] - WARN [Thread-10:LeaderElection@152] - Failed to register with JMX
javax.management.InstanceAlreadyExistsException: org.apache.ZooKeeperService:name0=LeaderElection
at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:453)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.internal_addObject(DefaultMBeanServerInterceptor.java:1484)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:963)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:917)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:312)
at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:483)
at org.apache.zookeeper.jmx.MBeanRegistry.register(MBeanRegistry.java:100)
at org.apache.zookeeper.server.quorum.LeaderElection.lookForLeader(LeaderElection.java:149)
at org.apache.zookeeper.test.LETest$LEThread.run(LETest.java:57)
2016-07-21 06:02:29,600 [myid:] - WARN [Thread-3:MBeanRegistry@104] - Failed to register MBean LeaderElection
2016-07-21 06:02:29,600 [myid:] - INFO [Thread-0:LeaderElection@187] - Server address: /127.0.0.1:11223
2016-07-21 06:02:29,600 [myid:] - WARN [Thread-5:MBeanRegistry@104] - Failed to register MBean LeaderElection
2016-07-21 06:02:29,601 [myid:] - INFO [Thread-0:LeaderElection@187] - Server address: /127.0.0.1:11225
2016-07-21 06:02:29,601 [myid:] - WARN [Thread-1:MBeanRegistry@104] - Failed to register MBean LeaderElection
2016-07-21 06:02:29,601 [myid:] - WARN [Thread-1:LeaderElection@152] - Failed to register with JMX
javax.management.InstanceAlreadyExistsException: org.apache.ZooKeeperService:name0=LeaderElection
at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:453)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.internal_addObject(DefaultMBeanServerInterceptor.java:1484)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:963)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:917)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:312)
at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:483)
at org.apache.zookeeper.jmx.MBeanRegistry.register(MBeanRegistry.java:100)
at org.apache.zookeeper.server.quorum.LeaderElection.lookForLeader(LeaderElection.java:149)
at org.apache.zookeeper.test.LETest$LEThread.run(LETest.java:57)
2016-07-21 06:02:29,600 [myid:] - WARN [Thread-4:MBeanRegistry@104] - Failed to register MBean LeaderElection
2016-07-21 06:02:29,602 [myid:] - WARN [Thread-4:LeaderElection@152] - Failed to register with JMX
javax.management.InstanceAlreadyExistsException: org.apache.ZooKeeperService:name0=LeaderElection
at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:453)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.internal_addObject(DefaultMBeanServerInterceptor.java:1484)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:963)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:917)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:312)
at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:483)
at org.apache.zookeeper.jmx.MBeanRegistry.register(MBeanRegistry.java:100)
at org.apache.zookeeper.server.quorum.LeaderElection.lookForLeader(LeaderElection.java:149)
at org.apache.zookeeper.test.LETest$LEThread.run(LETest.java:57)
2016-07-21 06:02:29,599 [myid:] - WARN [Thread-7:MBeanRegistry@104] - Failed to register MBean LeaderElection
2016-07-21 06:02:29,599 [myid:] - INFO [main:QuorumPeer@548] - acceptedEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
2016-07-21 06:02:29,599 [myid:] - WARN [Thread-6:MBeanRegistry@104] - Failed to register MBean LeaderElection
2016-07-21 06:02:29,604 [myid:] - WARN [Thread-6:LeaderElection@152] - Failed to register with JMX
javax.management.InstanceAlreadyExistsException: org.apache.ZooKeeperService:name0=LeaderElection
at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:453)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.internal_addObject(DefaultMBeanServerInterceptor.java:1484)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:963)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:917)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:312)
at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:483)
at org.apache.zookeeper.jmx.MBeanRegistry.register(MBeanRegistry.java:100)
at org.apache.zookeeper.server.quorum.LeaderElection.lookForLeader(LeaderElection.java:149)
at org.apache.zookeeper.test.LETest$LEThread.run(LETest.java:57)
2016-07-21 06:02:29,599 [myid:] - WARN [Thread-8:MBeanRegistry@104] - Failed to register MBean LeaderElection
2016-07-21 06:02:29,604 [myid:] - INFO [Thread-4:LeaderElection@187] - Server address: /127.0.0.1:11221
2016-07-21 06:02:29,598 [myid:] - WARN [Thread-12:LeaderElection@152] - Failed to register with JMX
javax.management.InstanceAlreadyExistsException: org.apache.ZooKeeperService:name0=LeaderElection
at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:453)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.internal_addObject(DefaultMBeanServerInterceptor.java:1484)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:963)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:917)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:312)
at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:483)
at org.apache.zookeeper.jmx.MBeanRegistry.register(MBeanRegistry.java:100)
at org.apache.zookeeper.server.quorum.LeaderElection.lookForLeader(LeaderElection.java:149)
at org.apache.zookeeper.test.LETest$LEThread.run(LETest.java:57)
2016-07-21 06:02:29,598 [myid:] - WARN [Thread-11:LeaderElection@152] - Failed to register with JMX
javax.management.InstanceAlreadyExistsException: org.apache.ZooKeeperService:name0=LeaderElection
at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:453)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.internal_addObject(DefaultMBeanServerInterceptor.java:1484)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:963)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:917)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:312)
at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:483)
at org.apache.zookeeper.jmx.MBeanRegistry.register(MBeanRegistry.java:100)
at org.apache.zookeeper.server.quorum.LeaderElection.lookForLeader(LeaderElection.java:149)
at org.apache.zookeeper.test.LETest$LEThread.run(LETest.java:57)
2016-07-21 06:02:29,598 [myid:] - WARN [Thread-9:LeaderElection@152] - Failed to register with JMX
javax.management.InstanceAlreadyExistsException: org.apache.ZooKeeperService:name0=LeaderElection
at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:453)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.internal_addObject(DefaultMBeanServerInterceptor.java:1484)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:963)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:917)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:312)
at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:483)
at org.apache.zookeeper.jmx.MBeanRegistry.register(MBeanRegistry.java:100)
at org.apache.zookeeper.server.quorum.LeaderElection.lookForLeader(LeaderElection.java:149)
at org.apache.zookeeper.test.LETest$LEThread.run(LETest.java:57)
2016-07-21 06:02:29,606 [myid:] - INFO [Thread-11:LeaderElection@187] - Server address: /127.0.0.1:11221
2016-07-21 06:02:29,606 [myid:] - INFO [Thread-9:LeaderElection@187] - Server address: /127.0.0.1:11221
2016-07-21 06:02:29,606 [myid:] - INFO [Thread-6:LeaderElection@187] - Server address: /127.0.0.1:11221
2016-07-21 06:02:29,606 [myid:] - INFO [Thread-9:LeaderElection@187] - Server address: /127.0.0.1:11223
2016-07-21 06:02:29,607 [myid:] - WARN [Thread-13:MBeanRegistry@104] - Failed to register MBean LeaderElection
2016-07-21 06:02:29,607 [myid:] - INFO [Thread-9:LeaderElection@187] - Server address: /127.0.0.1:11225
2016-07-21 06:02:29,605 [myid:] - INFO [Thread-12:LeaderElection@187] - Server address: /127.0.0.1:11221
2016-07-21 06:02:29,607 [myid:] - INFO [Thread-9:LeaderElection@187] - Server address: /127.0.0.1:11227
2016-07-21 06:02:29,605 [myid:] - INFO [Thread-4:LeaderElection@187] - Server address: /127.0.0.1:11223
2016-07-21 06:02:29,604 [myid:] - WARN [Thread-8:LeaderElection@152] - Failed to register with JMX
javax.management.InstanceAlreadyExistsException: org.apache.ZooKeeperService:name0=LeaderElection
at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:453)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.internal_addObject(DefaultMBeanServerInterceptor.java:1484)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:963)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:917)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:312)
at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:483)
at org.apache.zookeeper.jmx.MBeanRegistry.register(MBeanRegistry.java:100)
at org.apache.zookeeper.server.quorum.LeaderElection.lookForLeader(LeaderElection.java:149)
at org.apache.zookeeper.test.LETest$LEThread.run(LETest.java:57)
2016-07-21 06:02:29,603 [myid:] - WARN [Thread-7:LeaderElection@152] - Failed to register with JMX
javax.management.InstanceAlreadyExistsException: org.apache.ZooKeeperService:name0=LeaderElection
at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:453)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.internal_addObject(DefaultMBeanServerInterceptor.java:1484)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:963)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:917)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:312)
at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:483)
at org.apache.zookeeper.jmx.MBeanRegistry.register(MBeanRegistry.java:100)
at org.apache.zookeeper.server.quorum.LeaderElection.lookForLeader(LeaderElection.java:149)
at org.apache.zookeeper.test.LETest$LEThread.run(LETest.java:57)
2016-07-21 06:02:29,603 [myid:] - INFO [Thread-1:LeaderElection@187] - Server address: /127.0.0.1:11221
2016-07-21 06:02:29,601 [myid:] - INFO [Thread-0:LeaderElection@187] - Server address: /127.0.0.1:11227
2016-07-21 06:02:29,601 [myid:] - WARN [Thread-5:LeaderElection@152] - Failed to register with JMX
javax.management.InstanceAlreadyExistsException: org.apache.ZooKeeperService:name0=LeaderElection
at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:453)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.internal_addObject(DefaultMBeanServerInterceptor.java:1484)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:963)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:917)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:312)
at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:483)
at org.apache.zookeeper.jmx.MBeanRegistry.register(MBeanRegistry.java:100)
at org.apache.zookeeper.server.quorum.LeaderElection.lookForLeader(LeaderElection.java:149)
at org.apache.zookeeper.test.LETest$LEThread.run(LETest.java:57)
2016-07-21 06:02:29,601 [myid:] - INFO [Thread-10:LeaderElection@187] - Server address: /127.0.0.1:11221
2016-07-21 06:02:29,601 [myid:] - WARN [Thread-3:LeaderElection@152] - Failed to register with JMX
javax.management.InstanceAlreadyExistsException: org.apache.ZooKeeperService:name0=LeaderElection
at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:453)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.internal_addObject(DefaultMBeanServerInterceptor.java:1484)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:963)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:917)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:312)
at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:483)
at org.apache.zookeeper.jmx.MBeanRegistry.register(MBeanRegistry.java:100)
at org.apache.zookeeper.server.quorum.LeaderElection.lookForLeader(LeaderElection.java:149)
at org.apache.zookeeper.test.LETest$LEThread.run(LETest.java:57)
2016-07-21 06:02:29,610 [myid:] - INFO [Thread-10:LeaderElection@187] - Server address: /127.0.0.1:11223
2016-07-21 06:02:29,610 [myid:] - INFO [Thread-5:LeaderElection@187] - Server address: /127.0.0.1:11221
2016-07-21 06:02:29,611 [myid:] - INFO [Thread-10:LeaderElection@187] - Server address: /127.0.0.1:11225
2016-07-21 06:02:29,611 [myid:] - INFO [Thread-5:LeaderElection@187] - Server address: /127.0.0.1:11223
2016-07-21 06:02:29,609 [myid:] - INFO [Thread-0:LeaderElection@187] - Server address: /127.0.0.1:11229
2016-07-21 06:02:29,609 [myid:] - INFO [Thread-1:LeaderElection@187] - Server address: /127.0.0.1:11223
2016-07-21 06:02:29,609 [myid:] - INFO [Thread-7:LeaderElection@187] - Server address: /127.0.0.1:11221
2016-07-21 06:02:29,609 [myid:] - INFO [Thread-8:LeaderElection@187] - Server address: /127.0.0.1:11221
2016-07-21 06:02:29,608 [myid:] - INFO [Thread-4:LeaderElection@187] - Server address: /127.0.0.1:11225
2016-07-21 06:02:29,608 [myid:] - INFO [Thread-9:LeaderElection@187] - Server address: /127.0.0.1:11229
2016-07-21 06:02:29,613 [myid:] - INFO [Thread-4:LeaderElection@187] - Server address: /127.0.0.1:11227
2016-07-21 06:02:29,613 [myid:] - INFO [Thread-9:LeaderElection@187] - Server address: /127.0.0.1:11231
2016-07-21 06:02:29,607 [myid:] - INFO [Thread-12:LeaderElection@187] - Server address: /127.0.0.1:11223
2016-07-21 06:02:29,607 [myid:] - INFO [main:NIOServerCnxnFactory@89] - binding to port 0.0.0.0/0.0.0.0:11248
2016-07-21 06:02:29,607 [myid:] - WARN [Thread-13:LeaderElection@152] - Failed to register with JMX
javax.management.InstanceAlreadyExistsException: org.apache.ZooKeeperService:name0=LeaderElection
at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:453)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.internal_addObject(DefaultMBeanServerInterceptor.java:1484)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:963)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:917)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:312)
at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:483)
at org.apache.zookeeper.jmx.MBeanRegistry.register(MBeanRegistry.java:100)
at org.apache.zookeeper.server.quorum.LeaderElection.lookForLeader(LeaderElection.java:149)
at org.apache.zookeeper.test.LETest$LEThread.run(LETest.java:57)
2016-07-21 06:02:29,607 [myid:] - INFO [Thread-6:LeaderElection@187] - Server address: /127.0.0.1:11223
2016-07-21 06:02:29,606 [myid:] - INFO [Thread-11:LeaderElection@187] - Server address: /127.0.0.1:11223
2016-07-21 06:02:29,615 [myid:] - INFO [Thread-13:LeaderElection@187] - Server address: /127.0.0.1:11221
2016-07-21 06:02:29,614 [myid:] - INFO [Thread-12:LeaderElection@187] - Server address: /127.0.0.1:11225
2016-07-21 06:02:29,613 [myid:] - INFO [Thread-9:LeaderElection@187] - Server address: /127.0.0.1:11233
2016-07-21 06:02:29,615 [myid:] - INFO [Thread-12:LeaderElection@187] - Server address: /127.0.0.1:11227
2016-07-21 06:02:29,613 [myid:] - INFO [Thread-4:LeaderElection@187] - Server address: /127.0.0.1:11229
2016-07-21 06:02:29,613 [myid:] - INFO [Thread-8:LeaderElection@187] - Server address: /127.0.0.1:11223
2016-07-21 06:02:29,612 [myid:] - INFO [Thread-7:LeaderElection@187] - Server address: /127.0.0.1:11223
2016-07-21 06:02:29,612 [myid:] - INFO [Thread-1:LeaderElection@187] - Server address: /127.0.0.1:11225
2016-07-21 06:02:29,612 [myid:] - INFO [Thread-0:LeaderElection@187] - Server address: /127.0.0.1:11231
2016-07-21 06:02:29,612 [myid:] - INFO [Thread-5:LeaderElection@187] - Server address: /127.0.0.1:11225
2016-07-21 06:02:29,617 [myid:] - INFO [Thread-0:LeaderElection@187] - Server address: /127.0.0.1:11233
2016-07-21 06:02:29,617 [myid:] - INFO [Thread-5:LeaderElection@187] - Server address: /127.0.0.1:11227
2016-07-21 06:02:29,612 [myid:] - INFO [Thread-10:LeaderElection@187] - Server address: /127.0.0.1:11227
2016-07-21 06:02:29,617 [myid:] - INFO [Thread-5:LeaderElection@187] - Server address: /127.0.0.1:11229
2016-07-21 06:02:29,612 [myid:] - INFO [Thread-3:LeaderElection@187] - Server address: /127.0.0.1:11221
2016-07-21 06:02:29,618 [myid:] - INFO [Thread-5:LeaderElection@187] - Server address: /127.0.0.1:11231
2016-07-21 06:02:29,618 [myid:] - INFO [Thread-10:LeaderElection@187] - Server address: /127.0.0.1:11229
2016-07-21 06:02:29,618 [myid:] - INFO [Thread-5:LeaderElection@187] - Server address: /127.0.0.1:11233
2016-07-21 06:02:29,617 [myid:] - INFO [Thread-0:LeaderElection@187] - Server address: /127.0.0.1:11235
2016-07-21 06:02:29,618 [myid:] - INFO [Thread-5:LeaderElection@187] - Server address: /127.0.0.1:11235
2016-07-21 06:02:29,617 [myid:] - INFO [Thread-1:LeaderElection@187] - Server address: /127.0.0.1:11227
2016-07-21 06:02:29,619 [myid:] - INFO [Thread-5:LeaderElection@187] - Server address: /127.0.0.1:11237
2016-07-21 06:02:29,616 [myid:] - INFO [Thread-7:LeaderElection@187] - Server address: /127.0.0.1:11225
2016-07-21 06:02:29,619 [myid:] - INFO [Thread-1:LeaderElection@187] - Server address: /127.0.0.1:11229
2016-07-21 06:02:29,616 [myid:] - INFO [Thread-8:LeaderElection@187] - Server address: /127.0.0.1:11225
2016-07-21 06:02:29,616 [myid:] - INFO [Thread-4:LeaderElection@187] - Server address: /127.0.0.1:11231
2016-07-21 06:02:29,616 [myid:] - INFO [Thread-12:LeaderElection@187] - Server address: /127.0.0.1:11229
2016-07-21 06:02:29,619 [myid:] - INFO [Thread-4:LeaderElection@187] - Server address: /127.0.0.1:11233
2016-07-21 06:02:29,620 [myid:] - INFO [Thread-12:LeaderElection@187] - Server address: /127.0.0.1:11231
2016-07-21 06:02:29,620 [myid:] - INFO [Thread-4:LeaderElection@187] - Server address: /127.0.0.1:11235
2016-07-21 06:02:29,616 [myid:] - INFO [Thread-9:LeaderElection@187] - Server address: /127.0.0.1:11235
2016-07-21 06:02:29,615 [myid:] - INFO [Thread-13:LeaderElection@187] - Server address: /127.0.0.1:11223
2016-07-21 06:02:29,620 [myid:] - INFO [Thread-9:LeaderElection@187] - Server address: /127.0.0.1:11237
2016-07-21 06:02:29,615 [myid:] - INFO [Thread-11:LeaderElection@187] - Server address: /127.0.0.1:11225
2016-07-21 06:02:29,615 [myid:] - INFO [Thread-6:LeaderElection@187] - Server address: /127.0.0.1:11225
2016-07-21 06:02:29,615 [myid:] - INFO [main:QuorumPeer@533] - currentEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
2016-07-21 06:02:29,621 [myid:] - INFO [Thread-6:LeaderElection@187] - Server address: /127.0.0.1:11227
2016-07-21 06:02:29,621 [myid:] - INFO [Thread-11:LeaderElection@187] - Server address: /127.0.0.1:11227
2016-07-21 06:02:29,622 [myid:] - INFO [Thread-6:LeaderElection@187] - Server address: /127.0.0.1:11229
2016-07-21 06:02:29,621 [myid:] - INFO [Thread-9:LeaderElection@187] - Server address: /127.0.0.1:11239
2016-07-21 06:02:29,622 [myid:] - INFO [Thread-6:LeaderElection@187] - Server address: /127.0.0.1:11231
2016-07-21 06:02:29,621 [myid:] - INFO [Thread-13:LeaderElection@187] - Server address: /127.0.0.1:11225
2016-07-21 06:02:29,622 [myid:] - INFO [Thread-6:LeaderElection@187] - Server address: /127.0.0.1:11233
2016-07-21 06:02:29,620 [myid:] - INFO [Thread-4:LeaderElection@187] - Server address: /127.0.0.1:11237
2016-07-21 06:02:29,620 [myid:] - INFO [Thread-12:LeaderElection@187] - Server address: /127.0.0.1:11233
2016-07-21 06:02:29,623 [myid:] - INFO [Thread-4:LeaderElection@187] - Server address: /127.0.0.1:11239
2016-07-21 06:02:29,619 [myid:] - INFO [Thread-8:LeaderElection@187] - Server address: /127.0.0.1:11227
2016-07-21 06:02:29,623 [myid:] - INFO [Thread-4:LeaderElection@187] - Server address: /127.0.0.1:11241
2016-07-21 06:02:29,619 [myid:] - INFO [Thread-1:LeaderElection@187] - Server address: /127.0.0.1:11231
2016-07-21 06:02:29,623 [myid:] - INFO [Thread-8:LeaderElection@187] - Server address: /127.0.0.1:11229
2016-07-21 06:02:29,624 [myid:] - INFO [Thread-1:LeaderElection@187] - Server address: /127.0.0.1:11233
2016-07-21 06:02:29,624 [myid:] - INFO [Thread-8:LeaderElection@187] - Server address: /127.0.0.1:11231
2016-07-21 06:02:29,624 [myid:] - INFO [Thread-1:LeaderElection@187] - Server address: /127.0.0.1:11235
2016-07-21 06:02:29,624 [myid:] - INFO [Thread-8:LeaderElection@187] - Server address: /127.0.0.1:11233
2016-07-21 06:02:29,624 [myid:] - INFO [Thread-1:LeaderElection@187] - Server address: /127.0.0.1:11237
2016-07-21 06:02:29,619 [myid:] - INFO [Thread-7:LeaderElection@187] - Server address: /127.0.0.1:11227
2016-07-21 06:02:29,619 [myid:] - INFO [Thread-5:LeaderElection@187] - Server address: /127.0.0.1:11239
2016-07-21 06:02:29,618 [myid:] - INFO [Thread-0:LeaderElection@187] - Server address: /127.0.0.1:11237
2016-07-21 06:02:29,625 [myid:] - INFO [Thread-5:LeaderElection@187] - Server address: /127.0.0.1:11241
2016-07-21 06:02:29,618 [myid:] - INFO [Thread-10:LeaderElection@187] - Server address: /127.0.0.1:11231
2016-07-21 06:02:29,625 [myid:] - INFO [Thread-5:LeaderElection@187] - Server address: /127.0.0.1:11243
2016-07-21 06:02:29,618 [myid:] - INFO [Thread-3:LeaderElection@187] - Server address: /127.0.0.1:11223
2016-07-21 06:02:29,627 [myid:] - INFO [Thread-5:LeaderElection@187] - Server address: /127.0.0.1:11245
2016-07-21 06:02:29,627 [myid:] - INFO [Thread-3:LeaderElection@187] - Server address: /127.0.0.1:11225
2016-07-21 06:02:29,627 [myid:] - INFO [Thread-5:LeaderElection@187] - Server address: /127.0.0.1:11247
2016-07-21 06:02:29,627 [myid:] - INFO [Thread-3:LeaderElection@187] - Server address: /127.0.0.1:11227
2016-07-21 06:02:29,626 [myid:] - INFO [Thread-10:LeaderElection@187] - Server address: /127.0.0.1:11233
2016-07-21 06:02:29,625 [myid:] - INFO [Thread-0:LeaderElection@187] - Server address: /127.0.0.1:11239
2016-07-21 06:02:29,625 [myid:] - INFO [Thread-7:LeaderElection@187] - Server address: /127.0.0.1:11229
2016-07-21 06:02:29,625 [myid:] - INFO [Thread-1:LeaderElection@187] - Server address: /127.0.0.1:11239
2016-07-21 06:02:29,624 [myid:] - INFO [Thread-8:LeaderElection@187] - Server address: /127.0.0.1:11235
2016-07-21 06:02:29,628 [myid:] - INFO [Thread-1:LeaderElection@187] - Server address: /127.0.0.1:11241
2016-07-21 06:02:29,624 [myid:] - INFO [Thread-4:LeaderElection@187] - Server address: /127.0.0.1:11243
2016-07-21 06:02:29,623 [myid:] - INFO [Thread-12:LeaderElection@187] - Server address: /127.0.0.1:11235
2016-07-21 06:02:29,629 [myid:] - INFO [Thread-4:LeaderElection@187] - Server address: /127.0.0.1:11245
2016-07-21 06:02:29,629 [myid:] - INFO [Thread-12:LeaderElection@187] - Server address: /127.0.0.1:11237
2016-07-21 06:02:29,629 [myid:] - INFO [Thread-4:LeaderElection@187] - Server address: /127.0.0.1:11247
2016-07-21 06:02:29,623 [myid:] - INFO [Thread-6:LeaderElection@187] - Server address: /127.0.0.1:11235
2016-07-21 06:02:29,630 [myid:] - INFO [Thread-6:LeaderElection@187] - Server address: /127.0.0.1:11237
2016-07-21 06:02:29,630 [myid:] - INFO [Thread-6:LeaderElection@187] - Server address: /127.0.0.1:11239
2016-07-21 06:02:29,623 [myid:] - INFO [Thread-13:LeaderElection@187] - Server address: /127.0.0.1:11227
2016-07-21 06:02:29,630 [myid:] - INFO [Thread-6:LeaderElection@187] - Server address: /127.0.0.1:11241
2016-07-21 06:02:29,630 [myid:] - INFO [Thread-13:LeaderElection@187] - Server address: /127.0.0.1:11229
2016-07-21 06:02:29,622 [myid:] - INFO [main:QuorumPeer@548] - acceptedEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
2016-07-21 06:02:29,622 [myid:] - INFO [Thread-9:LeaderElection@187] - Server address: /127.0.0.1:11241
2016-07-21 06:02:29,622 [myid:] - INFO [Thread-11:LeaderElection@187] - Server address: /127.0.0.1:11229
2016-07-21 06:02:29,630 [myid:] - INFO [Thread-13:LeaderElection@187] - Server address: /127.0.0.1:11231
2016-07-21 06:02:29,629 [myid:] - INFO [Thread-12:LeaderElection@187] - Server address: /127.0.0.1:11239
2016-07-21 06:02:29,631 [myid:] - INFO [Thread-6:LeaderElection@187] - Server address: /127.0.0.1:11243
2016-07-21 06:02:29,629 [myid:] - INFO [Thread-1:LeaderElection@187] - Server address: /127.0.0.1:11243
2016-07-21 06:02:29,632 [myid:] - INFO [Thread-6:LeaderElection@187] - Server address: /127.0.0.1:11245
2016-07-21 06:02:29,628 [myid:] - INFO [Thread-8:LeaderElection@187] - Server address: /127.0.0.1:11237
2016-07-21 06:02:29,632 [myid:] - INFO [Thread-6:LeaderElection@187] - Server address: /127.0.0.1:11247
2016-07-21 06:02:29,632 [myid:] - INFO [Thread-8:LeaderElection@187] - Server address: /127.0.0.1:11239
2016-07-21 06:02:29,628 [myid:] - INFO [Thread-7:LeaderElection@187] - Server address: /127.0.0.1:11231
2016-07-21 06:02:29,628 [myid:] - INFO [Thread-0:LeaderElection@187] - Server address: /127.0.0.1:11241
2016-07-21 06:02:29,628 [myid:] - INFO [Thread-10:LeaderElection@187] - Server address: /127.0.0.1:11235
2016-07-21 06:02:29,633 [myid:] - INFO [Thread-0:LeaderElection@187] - Server address: /127.0.0.1:11243
2016-07-21 06:02:29,628 [myid:] - INFO [Thread-3:LeaderElection@187] - Server address: /127.0.0.1:11229
2016-07-21 06:02:29,633 [myid:] - INFO [Thread-0:LeaderElection@187] - Server address: /127.0.0.1:11245
2016-07-21 06:02:29,633 [myid:] - INFO [Thread-3:LeaderElection@187] - Server address: /127.0.0.1:11231
2016-07-21 06:02:29,633 [myid:] - INFO [Thread-10:LeaderElection@187] - Server address: /127.0.0.1:11237
2016-07-21 06:02:29,633 [myid:] - INFO [Thread-6:LeaderElection@187] - Server address: /127.0.0.1:11249
2016-07-21 06:02:29,633 [myid:] - INFO [Thread-7:LeaderElection@187] - Server address: /127.0.0.1:11233
2016-07-21 06:02:29,633 [myid:] - INFO [Thread-8:LeaderElection@187] - Server address: /127.0.0.1:11241
2016-07-21 06:02:29,632 [myid:] - INFO [Thread-1:LeaderElection@187] - Server address: /127.0.0.1:11245
2016-07-21 06:02:29,631 [myid:] - INFO [Thread-12:LeaderElection@187] - Server address: /127.0.0.1:11241
2016-07-21 06:02:29,634 [myid:] - INFO [Thread-1:LeaderElection@187] - Server address: /127.0.0.1:11247
2016-07-21 06:02:29,631 [myid:] - INFO [Thread-9:LeaderElection@187] - Server address: /127.0.0.1:11243
2016-07-21 06:02:29,635 [myid:] - INFO [Thread-12:LeaderElection@187] - Server address: /127.0.0.1:11243
2016-07-21 06:02:29,635 [myid:] - INFO [Thread-9:LeaderElection@187] - Server address: /127.0.0.1:11245
2016-07-21 06:02:29,631 [myid:] - INFO [Thread-13:LeaderElection@187] - Server address: /127.0.0.1:11233
2016-07-21 06:02:29,635 [myid:] - INFO [Thread-9:LeaderElection@187] - Server address: /127.0.0.1:11247
2016-07-21 06:02:29,631 [myid:] - INFO [Thread-11:LeaderElection@187] - Server address: /127.0.0.1:11231
2016-07-21 06:02:29,635 [myid:] - INFO [Thread-9:LeaderElection@187] - Server address: /127.0.0.1:11249
2016-07-21 06:02:29,635 [myid:] - INFO [Thread-13:LeaderElection@187] - Server address: /127.0.0.1:11235
2016-07-21 06:02:29,635 [myid:] - INFO [Thread-12:LeaderElection@187] - Server address: /127.0.0.1:11245
2016-07-21 06:02:29,635 [myid:] - INFO [Thread-1:LeaderElection@187] - Server address: /127.0.0.1:11249
2016-07-21 06:02:29,634 [myid:] - INFO [Thread-8:LeaderElection@187] - Server address: /127.0.0.1:11243
2016-07-21 06:02:29,634 [myid:] - INFO [Thread-7:LeaderElection@187] - Server address: /127.0.0.1:11235
2016-07-21 06:02:29,634 [myid:] - INFO [Thread-0:LeaderElection@187] - Server address: /127.0.0.1:11247
2016-07-21 06:02:29,634 [myid:] - INFO [Thread-10:LeaderElection@187] - Server address: /127.0.0.1:11239
2016-07-21 06:02:29,634 [myid:] - INFO [main:NIOServerCnxnFactory@89] - binding to port 0.0.0.0/0.0.0.0:11250
2016-07-21 06:02:29,634 [myid:] - INFO [Thread-3:LeaderElection@187] - Server address: /127.0.0.1:11233
2016-07-21 06:02:29,633 [myid:] - WARN [Thread-14:MBeanRegistry@104] - Failed to register MBean LeaderElection
2016-07-21 06:02:29,637 [myid:] - INFO [Thread-3:LeaderElection@187] - Server address: /127.0.0.1:11235
2016-07-21 06:02:29,637 [myid:] - INFO [Thread-10:LeaderElection@187] - Server address: /127.0.0.1:11241
2016-07-21 06:02:29,637 [myid:] - INFO [Thread-0:LeaderElection@187] - Server address: /127.0.0.1:11249
2016-07-21 06:02:29,636 [myid:] - INFO [Thread-7:LeaderElection@187] - Server address: /127.0.0.1:11237
2016-07-21 06:02:29,636 [myid:] - INFO [Thread-8:LeaderElection@187] - Server address: /127.0.0.1:11245
2016-07-21 06:02:29,636 [myid:] - INFO [Thread-12:LeaderElection@187] - Server address: /127.0.0.1:11247
2016-07-21 06:02:29,638 [myid:] - INFO [Thread-8:LeaderElection@187] - Server address: /127.0.0.1:11247
2016-07-21 06:02:29,636 [myid:] - INFO [Thread-13:LeaderElection@187] - Server address: /127.0.0.1:11237
2016-07-21 06:02:29,638 [myid:] - INFO [Thread-8:LeaderElection@187] - Server address: /127.0.0.1:11249
2016-07-21 06:02:29,638 [myid:] - INFO [Thread-13:LeaderElection@187] - Server address: /127.0.0.1:11239
2016-07-21 06:02:29,636 [myid:] - INFO [Thread-11:LeaderElection@187] - Server address: /127.0.0.1:11233
2016-07-21 06:02:29,638 [myid:] - INFO [Thread-12:LeaderElection@187] - Server address: /127.0.0.1:11249
2016-07-21 06:02:29,638 [myid:] - INFO [Thread-7:LeaderElection@187] - Server address: /127.0.0.1:11239
2016-07-21 06:02:29,637 [myid:] - INFO [Thread-10:LeaderElection@187] - Server address: /127.0.0.1:11243
2016-07-21 06:02:29,637 [myid:] - INFO [main:QuorumPeer@533] - currentEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
2016-07-21 06:02:29,637 [myid:] - INFO [Thread-3:LeaderElection@187] - Server address: /127.0.0.1:11237
2016-07-21 06:02:29,637 [myid:] - WARN [Thread-14:LeaderElection@152] - Failed to register with JMX
javax.management.InstanceAlreadyExistsException: org.apache.ZooKeeperService:name0=LeaderElection
at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:453)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.internal_addObject(DefaultMBeanServerInterceptor.java:1484)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:963)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:917)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:312)
at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:483)
at org.apache.zookeeper.jmx.MBeanRegistry.register(MBeanRegistry.java:100)
at org.apache.zookeeper.server.quorum.LeaderElection.lookForLeader(LeaderElection.java:149)
at org.apache.zookeeper.test.LETest$LEThread.run(LETest.java:57)
2016-07-21 06:02:29,639 [myid:] - INFO [Thread-3:LeaderElection@187] - Server address: /127.0.0.1:11239
2016-07-21 06:02:29,640 [myid:] - INFO [Thread-14:LeaderElection@187] - Server address: /127.0.0.1:11221
2016-07-21 06:02:29,640 [myid:] - INFO [Thread-3:LeaderElection@187] - Server address: /127.0.0.1:11241
2016-07-21 06:02:29,639 [myid:] - INFO [Thread-10:LeaderElection@187] - Server address: /127.0.0.1:11245
2016-07-21 06:02:29,639 [myid:] - INFO [Thread-7:LeaderElection@187] - Server address: /127.0.0.1:11241
2016-07-21 06:02:29,639 [myid:] - INFO [Thread-11:LeaderElection@187] - Server address: /127.0.0.1:11235
2016-07-21 06:02:29,639 [myid:] - INFO [Thread-13:LeaderElection@187] - Server address: /127.0.0.1:11241
2016-07-21 06:02:29,641 [myid:] - INFO [Thread-11:LeaderElection@187] - Server add
...[truncated 187743 chars]...
> 1
2016-07-21 06:02:36,006 [myid:] - INFO [Thread-13:LeaderElection@187] - Server address: /127.0.0.1:11233
2016-07-21 06:02:36,006 [myid:] - INFO [Thread-18:LeaderElection@130] - 21 -> 1
2016-07-21 06:02:36,008 [myid:] - INFO [Thread-13:LeaderElection@187] - Server address: /127.0.0.1:11235
2016-07-21 06:02:36,006 [myid:] - INFO [Thread-3:LeaderElection@187] - Server address: /127.0.0.1:11223
2016-07-21 06:02:35,931 [myid:] - INFO [Thread-10:LeaderElection@187] - Server address: /127.0.0.1:11223
2016-07-21 06:02:35,930 [myid:] - INFO [Thread-9:LeaderElection@187] - Server address: /127.0.0.1:11223
2016-07-21 06:02:35,930 [myid:] - INFO [Thread-15:LeaderElection@187] - Server address: /127.0.0.1:11223
2016-07-21 06:02:35,930 [myid:] - INFO [Thread-7:LeaderElection@187] - Server address: /127.0.0.1:11235
2016-07-21 06:02:35,930 [myid:] - INFO [Thread-0:LeaderElection@187] - Server address: /127.0.0.1:11243
2016-07-21 06:02:36,008 [myid:] - INFO [Thread-7:LeaderElection@187] - Server address: /127.0.0.1:11237
2016-07-21 06:02:35,930 [myid:] - INFO [Thread-1:LeaderElection@187] - Server address: /127.0.0.1:11239
2016-07-21 06:02:35,930 [myid:] - INFO [Thread-17:LeaderElection@187] - Server address: /127.0.0.1:11235
2016-07-21 06:02:35,929 [myid:] - INFO [Thread-5:LeaderElection@187] - Server address: /127.0.0.1:11241
2016-07-21 06:02:35,929 [myid:] - INFO [Thread-4:LeaderElection@187] - Server address: /127.0.0.1:11255
2016-07-21 06:02:35,929 [myid:] - INFO [Thread-11:LeaderElection@187] - Server address: /127.0.0.1:11251
2016-07-21 06:02:36,009 [myid:] - INFO [Thread-4:LeaderElection@187] - Server address: /127.0.0.1:11253
2016-07-21 06:02:35,927 [myid:] - INFO [Thread-6:LeaderElection@187] - Server address: /127.0.0.1:11239
2016-07-21 06:02:35,927 [myid:] - INFO [Thread-27:LeaderElection@187] - Server address: /127.0.0.1:11249
2016-07-21 06:02:36,010 [myid:] - INFO [Thread-6:LeaderElection@187] - Server address: /127.0.0.1:11241
2016-07-21 06:02:35,927 [myid:] - INFO [Thread-8:LeaderElection@187] - Server address: /127.0.0.1:11235
2016-07-21 06:02:35,926 [myid:] - INFO [Thread-28:LeaderElection@187] - Server address: /127.0.0.1:11237
2016-07-21 06:02:35,926 [myid:] - INFO [Thread-30:LeaderElection@187] - Server address: /127.0.0.1:11239
2016-07-21 06:02:35,926 [myid:] - INFO [Thread-29:LeaderElection@187] - Server address: /127.0.0.1:11241
2016-07-21 06:02:35,926 [myid:] - INFO [Thread-26:LeaderElection@130] - 2 -> 1
2016-07-21 06:02:35,925 [myid:] - INFO [Thread-16:LeaderElection@130] - 24 -> 1
2016-07-21 06:02:36,011 [myid:] - INFO [Thread-16:LeaderElection@130] - 16 -> 1
2016-07-21 06:02:36,011 [myid:] - INFO [Thread-16:LeaderElection@130] - 23 -> 1
2016-07-21 06:02:35,925 [myid:] - INFO [Thread-20:LeaderElection@130] - 13 -> 1
2016-07-21 06:02:36,012 [myid:] - INFO [Thread-20:LeaderElection@130] - 24 -> 1
2016-07-21 06:02:36,012 [myid:] - INFO [Thread-20:LeaderElection@130] - 16 -> 1
2016-07-21 06:02:35,924 [myid:] - INFO [Thread-25:LeaderElection@130] - 9 -> 1
2016-07-21 06:02:35,924 [myid:] - INFO [Thread-21:LeaderElection@130] - 9 -> 1
2016-07-21 06:02:36,012 [myid:] - INFO [Thread-21:LeaderElection@130] - 10 -> 1
2016-07-21 06:02:36,012 [myid:] - INFO [Thread-21:LeaderElection@130] - 0 -> 1
2016-07-21 06:02:36,012 [myid:] - INFO [Thread-21:LeaderElection@130] - 4 -> 1
2016-07-21 06:02:35,924 [myid:] - INFO [Thread-23:LeaderElection@130] - 13 -> 1
2016-07-21 06:02:36,012 [myid:] - INFO [Thread-23:LeaderElection@130] - 21 -> 1
2016-07-21 06:02:36,013 [myid:] - INFO [Thread-23:LeaderElection@130] - 24 -> 1
2016-07-21 06:02:36,013 [myid:] - INFO [Thread-23:LeaderElection@130] - 16 -> 1
2016-07-21 06:02:36,013 [myid:] - INFO [Thread-23:LeaderElection@130] - 23 -> 1
2016-07-21 06:02:35,924 [myid:] - INFO [Thread-19:LeaderElection@130] - 3 -> 1
2016-07-21 06:02:35,924 [myid:] - INFO [Thread-12:LeaderElection@130] - 20 -> 1
2016-07-21 06:02:36,013 [myid:] - INFO [Thread-19:LeaderElection@130] - 2 -> 1
2016-07-21 06:02:36,012 [myid:] - INFO [Thread-21:LeaderElection@130] - 3 -> 1
2016-07-21 06:02:36,012 [myid:] - INFO [Thread-25:LeaderElection@130] - 10 -> 1
2016-07-21 06:02:36,013 [myid:] - INFO [Thread-25:LeaderElection@130] - 0 -> 1
2016-07-21 06:02:36,013 [myid:] - INFO [Thread-25:LeaderElection@130] - 4 -> 1
2016-07-21 06:02:36,013 [myid:] - INFO [Thread-25:LeaderElection@130] - 3 -> 1
2016-07-21 06:02:36,014 [myid:] - INFO [Thread-25:LeaderElection@130] - 2 -> 1
2016-07-21 06:02:36,014 [myid:] - INFO [Thread-25:LeaderElection@130] - 1 -> 1
2016-07-21 06:02:36,014 [myid:] - INFO [Thread-25:LeaderElection@130] - 7 -> 1
2016-07-21 06:02:36,014 [myid:] - INFO [Thread-25:LeaderElection@130] - 18 -> 1
2016-07-21 06:02:36,014 [myid:] - INFO [Thread-25:LeaderElection@130] - 17 -> 1
2016-07-21 06:02:36,014 [myid:] - INFO [Thread-25:LeaderElection@130] - 25 -> 1
2016-07-21 06:02:36,014 [myid:] - INFO [Thread-25:LeaderElection@130] - 20 -> 1
2016-07-21 06:02:36,011 [myid:] - INFO [Thread-29:LeaderElection@187] - Server address: /127.0.0.1:11243
2016-07-21 06:02:36,011 [myid:] - INFO [Thread-26:LeaderElection@130] - 1 -> 1
2016-07-21 06:02:36,011 [myid:] - INFO [Thread-30:LeaderElection@187] - Server address: /127.0.0.1:11241
2016-07-21 06:02:36,011 [myid:] - INFO [Thread-28:LeaderElection@187] - Server address: /127.0.0.1:11239
2016-07-21 06:02:36,011 [myid:] - INFO [Thread-8:LeaderElection@187] - Server address: /127.0.0.1:11237
2016-07-21 06:02:36,010 [myid:] - INFO [Thread-6:LeaderElection@187] - Server address: /127.0.0.1:11243
2016-07-21 06:02:36,015 [myid:] - INFO [Thread-8:LeaderElection@187] - Server address: /127.0.0.1:11239
2016-07-21 06:02:36,010 [myid:] - INFO [Thread-27:LeaderElection@187] - Server address: /127.0.0.1:11251
2016-07-21 06:02:36,010 [myid:] - INFO [Thread-4:LeaderElection@187] - Server address: /127.0.0.1:11259
2016-07-21 06:02:36,010 [myid:] - INFO [Thread-11:LeaderElection@187] - Server address: /127.0.0.1:11255
2016-07-21 06:02:36,009 [myid:] - INFO [Thread-5:LeaderElection@187] - Server address: /127.0.0.1:11243
2016-07-21 06:02:36,009 [myid:] - INFO [Thread-17:LeaderElection@187] - Server address: /127.0.0.1:11237
2016-07-21 06:02:36,009 [myid:] - INFO [Thread-1:LeaderElection@187] - Server address: /127.0.0.1:11241
2016-07-21 06:02:36,009 [myid:] - INFO [Thread-7:LeaderElection@187] - Server address: /127.0.0.1:11239
2016-07-21 06:02:36,009 [myid:] - INFO [Thread-0:LeaderElection@187] - Server address: /127.0.0.1:11245
2016-07-21 06:02:36,017 [myid:] - INFO [Thread-7:LeaderElection@187] - Server address: /127.0.0.1:11241
2016-07-21 06:02:36,008 [myid:] - INFO [Thread-15:LeaderElection@187] - Server address: /127.0.0.1:11225
2016-07-21 06:02:36,008 [myid:] - INFO [Thread-9:LeaderElection@187] - Server address: /127.0.0.1:11225
2016-07-21 06:02:36,008 [myid:] - INFO [Thread-10:LeaderElection@187] - Server address: /127.0.0.1:11225
2016-07-21 06:02:36,008 [myid:] - INFO [Thread-3:LeaderElection@187] - Server address: /127.0.0.1:11225
2016-07-21 06:02:36,008 [myid:] - INFO [Thread-13:LeaderElection@187] - Server address: /127.0.0.1:11237
2016-07-21 06:02:36,008 [myid:] - INFO [Thread-18:LeaderElection@130] - 24 -> 1
2016-07-21 06:02:36,051 [myid:] - INFO [Thread-18:LeaderElection@130] - 16 -> 1
2016-07-21 06:02:36,007 [myid:] - INFO [Thread-24:LeaderElection@130] - 21 -> 1
2016-07-21 06:02:36,051 [myid:] - INFO [Thread-24:LeaderElection@130] - 16 -> 1
2016-07-21 06:02:36,051 [myid:] - INFO [Thread-24:LeaderElection@130] - 24 -> 1
2016-07-21 06:02:36,007 [myid:] - INFO [Thread-14:LeaderElection@130] - 4 -> 1
2016-07-21 06:02:36,051 [myid:] - INFO [Thread-14:LeaderElection@130] - 3 -> 1
2016-07-21 06:02:36,051 [myid:] - INFO [Thread-14:LeaderElection@130] - 2 -> 1
2016-07-21 06:02:36,051 [myid:] - INFO [Thread-14:LeaderElection@130] - 1 -> 1
2016-07-21 06:02:36,052 [myid:] - INFO [Thread-14:LeaderElection@130] - 7 -> 1
2016-07-21 06:02:36,052 [myid:] - INFO [Thread-14:LeaderElection@130] - 18 -> 1
2016-07-21 06:02:36,052 [myid:] - INFO [Thread-14:LeaderElection@130] - 17 -> 1
2016-07-21 06:02:36,052 [myid:] - INFO [Thread-14:LeaderElection@130] - 25 -> 1
2016-07-21 06:02:36,007 [myid:] - INFO [Thread-22:LeaderElection@130] - 6 -> 1
2016-07-21 06:02:36,052 [myid:] - INFO [Thread-22:LeaderElection@130] - 11 -> 1
2016-07-21 06:02:36,052 [myid:] - INFO [Thread-14:LeaderElection@130] - 20 -> 1
2016-07-21 06:02:36,051 [myid:] - INFO [Thread-18:LeaderElection@130] - 23 -> 1
2016-07-21 06:02:36,051 [myid:] - INFO [Thread-13:LeaderElection@187] - Server address: /127.0.0.1:11239
2016-07-21 06:02:36,051 [myid:] - INFO [Thread-3:LeaderElection@187] - Server address: /127.0.0.1:11227
2016-07-21 06:02:36,050 [myid:] - INFO [Thread-9:LeaderElection@187] - Server address: /127.0.0.1:11227
2016-07-21 06:02:36,050 [myid:] - INFO [Thread-10:LeaderElection@187] - Server address: /127.0.0.1:11227
2016-07-21 06:02:36,050 [myid:] - INFO [Thread-15:LeaderElection@187] - Server address: /127.0.0.1:11227
2016-07-21 06:02:36,017 [myid:] - INFO [Thread-7:LeaderElection@187] - Server address: /127.0.0.1:11243
2016-07-21 06:02:36,017 [myid:] - INFO [Thread-0:LeaderElection@187] - Server address: /127.0.0.1:11247
2016-07-21 06:02:36,053 [myid:] - INFO [Thread-7:LeaderElection@187] - Server address: /127.0.0.1:11245
2016-07-21 06:02:36,053 [myid:] - INFO [Thread-0:LeaderElection@187] - Server address: /127.0.0.1:11249
2016-07-21 06:02:36,053 [myid:] - INFO [Thread-7:LeaderElection@187] - Server address: /127.0.0.1:11247
2016-07-21 06:02:36,053 [myid:] - INFO [Thread-0:LeaderElection@187] - Server address: /127.0.0.1:11251
2016-07-21 06:02:36,016 [myid:] - INFO [Thread-1:LeaderElection@187] - Server address: /127.0.0.1:11243
2016-07-21 06:02:36,016 [myid:] - INFO [Thread-17:LeaderElection@187] - Server address: /127.0.0.1:11239
2016-07-21 06:02:36,016 [myid:] - INFO [Thread-5:LeaderElection@187] - Server address: /127.0.0.1:11245
2016-07-21 06:02:36,016 [myid:] - INFO [Thread-11:LeaderElection@187] - Server address: /127.0.0.1:11253
2016-07-21 06:02:36,016 [myid:] - INFO [Thread-27:LeaderElection@187] - Server address: /127.0.0.1:11255
2016-07-21 06:02:36,016 [myid:] - INFO [Thread-4:LeaderElection@187] - Server address: /127.0.0.1:11257
2016-07-21 06:02:36,054 [myid:] - INFO [Thread-27:LeaderElection@187] - Server address: /127.0.0.1:11253
2016-07-21 06:02:36,015 [myid:] - INFO [Thread-8:LeaderElection@187] - Server address: /127.0.0.1:11241
2016-07-21 06:02:36,054 [myid:] - INFO [Thread-27:LeaderElection@187] - Server address: /127.0.0.1:11259
2016-07-21 06:02:36,015 [myid:] - INFO [Thread-6:LeaderElection@187] - Server address: /127.0.0.1:11245
2016-07-21 06:02:36,055 [myid:] - INFO [Thread-27:LeaderElection@187] - Server address: /127.0.0.1:11257
2016-07-21 06:02:36,015 [myid:] - INFO [Thread-28:LeaderElection@187] - Server address: /127.0.0.1:11241
2016-07-21 06:02:36,055 [myid:] - INFO [Thread-27:LeaderElection@187] - Server address: /127.0.0.1:11263
2016-07-21 06:02:36,015 [myid:] - INFO [Thread-30:LeaderElection@187] - Server address: /127.0.0.1:11243
2016-07-21 06:02:36,055 [myid:] - INFO [Thread-27:LeaderElection@187] - Server address: /127.0.0.1:11261
2016-07-21 06:02:36,015 [myid:] - INFO [Thread-29:LeaderElection@187] - Server address: /127.0.0.1:11245
2016-07-21 06:02:36,055 [myid:] - INFO [Thread-27:LeaderElection@187] - Server address: /127.0.0.1:11267
2016-07-21 06:02:36,014 [myid:] - INFO [Thread-26:LeaderElection@130] - 26 -> 1
2016-07-21 06:02:36,055 [myid:] - INFO [Thread-26:LeaderElection@130] - 18 -> 1
2016-07-21 06:02:36,055 [myid:] - INFO [Thread-26:LeaderElection@130] - 17 -> 1
2016-07-21 06:02:36,055 [myid:] - INFO [Thread-26:LeaderElection@130] - 25 -> 1
2016-07-21 06:02:36,056 [myid:] - INFO [Thread-26:LeaderElection@130] - 20 -> 1
2016-07-21 06:02:36,056 [myid:] - INFO [Thread-26:LeaderElection@130] - 27 -> 1
2016-07-21 06:02:36,056 [myid:] - INFO [Thread-26:LeaderElection@130] - 19 -> 1
2016-07-21 06:02:36,056 [myid:] - INFO [Thread-26:LeaderElection@130] - 14 -> 1
2016-07-21 06:02:36,056 [myid:] - INFO [Thread-26:LeaderElection@130] - 22 -> 1
2016-07-21 06:02:36,014 [myid:] - INFO [Thread-25:LeaderElection@130] - 19 -> 1
2016-07-21 06:02:36,013 [myid:] - INFO [Thread-21:LeaderElection@130] - 2 -> 1
2016-07-21 06:02:36,013 [myid:] - INFO [Thread-19:LeaderElection@130] - 1 -> 1
2016-07-21 06:02:36,013 [myid:] - INFO [Thread-12:LeaderElection@130] - 19 -> 1
2016-07-21 06:02:36,056 [myid:] - INFO [Thread-19:LeaderElection@130] - 7 -> 1
2016-07-21 06:02:36,056 [myid:] - INFO [Thread-21:LeaderElection@130] - 1 -> 1
2016-07-21 06:02:36,056 [myid:] - INFO [Thread-25:LeaderElection@130] - 14 -> 1
2016-07-21 06:02:36,056 [myid:] - INFO [Thread-26:LeaderElection@130] - 13 -> 1
2016-07-21 06:02:36,055 [myid:] - INFO [Thread-27:LeaderElection@187] - Server address: /127.0.0.1:11265
2016-07-21 06:02:36,055 [myid:] - INFO [Thread-29:LeaderElection@187] - Server address: /127.0.0.1:11247
2016-07-21 06:02:36,055 [myid:] - INFO [Thread-30:LeaderElection@187] - Server address: /127.0.0.1:11245
2016-07-21 06:02:36,057 [myid:] - INFO [Thread-29:LeaderElection@187] - Server address: /127.0.0.1:11249
2016-07-21 06:02:36,055 [myid:] - INFO [Thread-28:LeaderElection@187] - Server address: /127.0.0.1:11243
2016-07-21 06:02:36,057 [myid:] - INFO [Thread-29:LeaderElection@187] - Server address: /127.0.0.1:11251
2016-07-21 06:02:36,057 [myid:] - INFO [Thread-28:LeaderElection@187] - Server address: /127.0.0.1:11245
2016-07-21 06:02:36,058 [myid:] - INFO [Thread-29:LeaderElection@187] - Server address: /127.0.0.1:11255
2016-07-21 06:02:36,055 [myid:] - INFO [Thread-6:LeaderElection@187] - Server address: /127.0.0.1:11247
2016-07-21 06:02:36,055 [myid:] - INFO [Thread-8:LeaderElection@187] - Server address: /127.0.0.1:11243
2016-07-21 06:02:36,054 [myid:] - INFO [Thread-4:LeaderElection@187] - Server address: /127.0.0.1:11263
2016-07-21 06:02:36,054 [myid:] - INFO [Thread-11:LeaderElection@187] - Server address: /127.0.0.1:11259
2016-07-21 06:02:36,058 [myid:] - INFO [Thread-4:LeaderElection@187] - Server address: /127.0.0.1:11261
2016-07-21 06:02:36,054 [myid:] - INFO [Thread-5:LeaderElection@187] - Server address: /127.0.0.1:11247
2016-07-21 06:02:36,054 [myid:] - INFO [Thread-17:LeaderElection@187] - Server address: /127.0.0.1:11241
2016-07-21 06:02:36,054 [myid:] - INFO [Thread-1:LeaderElection@187] - Server address: /127.0.0.1:11245
2016-07-21 06:02:36,054 [myid:] - INFO [Thread-0:LeaderElection@187] - Server address: /127.0.0.1:11255
2016-07-21 06:02:36,053 [myid:] - INFO [Thread-7:LeaderElection@187] - Server address: /127.0.0.1:11249
2016-07-21 06:02:36,059 [myid:] - INFO [Thread-0:LeaderElection@187] - Server address: /127.0.0.1:11253
2016-07-21 06:02:36,053 [myid:] - INFO [Thread-15:LeaderElection@187] - Server address: /127.0.0.1:11229
2016-07-21 06:02:36,059 [myid:] - INFO [Thread-0:LeaderElection@187] - Server address: /127.0.0.1:11259
2016-07-21 06:02:36,059 [myid:] - INFO [Thread-15:LeaderElection@187] - Server address: /127.0.0.1:11231
2016-07-21 06:02:36,053 [myid:] - INFO [Thread-10:LeaderElection@187] - Server address: /127.0.0.1:11229
2016-07-21 06:02:36,053 [myid:] - INFO [Thread-9:LeaderElection@187] - Server address: /127.0.0.1:11229
2016-07-21 06:02:36,053 [myid:] - INFO [Thread-3:LeaderElection@187] - Server address: /127.0.0.1:11229
2016-07-21 06:02:36,053 [myid:] - INFO [Thread-13:LeaderElection@187] - Server address: /127.0.0.1:11241
2016-07-21 06:02:36,060 [myid:] - INFO [Thread-3:LeaderElection@187] - Server address: /127.0.0.1:11231
2016-07-21 06:02:36,052 [myid:] - INFO [Thread-14:LeaderElection@130] - 19 -> 1
2016-07-21 06:02:36,060 [myid:] - INFO [Thread-14:LeaderElection@130] - 14 -> 1
2016-07-21 06:02:36,052 [myid:] - INFO [Thread-22:LeaderElection@130] - 12 -> 1
2016-07-21 06:02:36,060 [myid:] - INFO [Thread-22:LeaderElection@130] - 9 -> 1
2016-07-21 06:02:36,060 [myid:] - INFO [Thread-22:LeaderElection@130] - 10 -> 1
2016-07-21 06:02:36,060 [myid:] - INFO [Thread-22:LeaderElection@130] - 0 -> 1
2016-07-21 06:02:36,060 [myid:] - INFO [Thread-22:LeaderElection@130] - 4 -> 1
2016-07-21 06:02:36,060 [myid:] - INFO [Thread-22:LeaderElection@130] - 3 -> 1
2016-07-21 06:02:36,060 [myid:] - INFO [Thread-22:LeaderElection@130] - 2 -> 1
2016-07-21 06:02:36,060 [myid:] - INFO [Thread-22:LeaderElection@130] - 1 -> 1
2016-07-21 06:02:36,060 [myid:] - INFO [Thread-14:LeaderElection@130] - 22 -> 1
2016-07-21 06:02:36,061 [myid:] - INFO [Thread-14:LeaderElection@130] - 21 -> 1
2016-07-21 06:02:36,061 [myid:] - INFO [Thread-14:LeaderElection@130] - 13 -> 1
2016-07-21 06:02:36,061 [myid:] - INFO [Thread-14:LeaderElection@130] - 24 -> 1
2016-07-21 06:02:36,061 [myid:] - INFO [Thread-14:LeaderElection@130] - 16 -> 1
2016-07-21 06:02:36,060 [myid:] - INFO [Thread-3:LeaderElection@187] - Server address: /127.0.0.1:11233
2016-07-21 06:02:36,060 [myid:] - INFO [Thread-13:LeaderElection@187] - Server address: /127.0.0.1:11243
2016-07-21 06:02:36,059 [myid:] - INFO [Thread-9:LeaderElection@187] - Server address: /127.0.0.1:11231
2016-07-21 06:02:36,059 [myid:] - INFO [Thread-10:LeaderElection@187] - Server address: /127.0.0.1:11231
2016-07-21 06:02:36,061 [myid:] - INFO [Thread-9:LeaderElection@187] - Server address: /127.0.0.1:11233
2016-07-21 06:02:36,062 [myid:] - INFO [Thread-10:LeaderElection@187] - Server address: /127.0.0.1:11233
2016-07-21 06:02:36,059 [myid:] - INFO [Thread-15:LeaderElection@187] - Server address: /127.0.0.1:11233
2016-07-21 06:02:36,059 [myid:] - INFO [Thread-0:LeaderElection@187] - Server address: /127.0.0.1:11257
2016-07-21 06:02:36,059 [myid:] - INFO [Thread-7:LeaderElection@187] - Server address: /127.0.0.1:11251
2016-07-21 06:02:36,062 [myid:] - INFO [Thread-0:LeaderElection@187] - Server address: /127.0.0.1:11263
2016-07-21 06:02:36,059 [myid:] - INFO [Thread-1:LeaderElection@187] - Server address: /127.0.0.1:11247
2016-07-21 06:02:36,059 [myid:] - INFO [Thread-17:LeaderElection@187] - Server address: /127.0.0.1:11243
2016-07-21 06:02:36,062 [myid:] - INFO [Thread-1:LeaderElection@187] - Server address: /127.0.0.1:11249
2016-07-21 06:02:36,058 [myid:] - INFO [Thread-5:LeaderElection@187] - Server address: /127.0.0.1:11249
2016-07-21 06:02:36,058 [myid:] - INFO [Thread-4:LeaderElection@187] - Server address: /127.0.0.1:11267
2016-07-21 06:02:36,063 [myid:] - INFO [Thread-5:LeaderElection@187] - Server address: /127.0.0.1:11251
2016-07-21 06:02:36,058 [myid:] - INFO [Thread-11:LeaderElection@187] - Server address: /127.0.0.1:11257
2016-07-21 06:02:36,058 [myid:] - INFO [Thread-8:LeaderElection@187] - Server address: /127.0.0.1:11245
2016-07-21 06:02:36,058 [myid:] - INFO [Thread-6:LeaderElection@187] - Server address: /127.0.0.1:11249
2016-07-21 06:02:36,058 [myid:] - INFO [Thread-29:LeaderElection@187] - Server address: /127.0.0.1:11253
2016-07-21 06:02:36,063 [myid:] - INFO [Thread-6:LeaderElection@187] - Server address: /127.0.0.1:11251
2016-07-21 06:02:36,063 [myid:] - INFO [Thread-29:LeaderElection@187] - Server address: /127.0.0.1:11259
2016-07-21 06:02:36,063 [myid:] - INFO [Thread-6:LeaderElection@187] - Server address: /127.0.0.1:11255
2016-07-21 06:02:36,058 [myid:] - INFO [Thread-28:LeaderElection@187] - Server address: /127.0.0.1:11247
2016-07-21 06:02:36,057 [myid:] - INFO [Thread-30:LeaderElection@187] - Server address: /127.0.0.1:11247
2016-07-21 06:02:36,057 [myid:] - INFO [Thread-27:LeaderElection@187] - Server address: /127.0.0.1:11271
2016-07-21 06:02:36,057 [myid:] - INFO [Thread-26:LeaderElection@130] - 21 -> 1
2016-07-21 06:02:36,057 [myid:] - INFO [Thread-25:LeaderElection@130] - 22 -> 1
2016-07-21 06:02:36,057 [myid:] - INFO [Thread-21:LeaderElection@130] - 7 -> 1
2016-07-21 06:02:36,064 [myid:] - INFO [Thread-21:LeaderElection@130] - 18 -> 1
2016-07-21 06:02:36,064 [myid:] - INFO [Thread-21:LeaderElection@130] - 17 -> 1
2016-07-21 06:02:36,065 [myid:] - INFO [Thread-21:LeaderElection@130] - 25 -> 1
2016-07-21 06:02:36,065 [myid:] - INFO [Thread-21:LeaderElection@130] - 20 -> 1
2016-07-21 06:02:36,065 [myid:] - INFO [Thread-21:LeaderElection@130] - 19 -> 1
2016-07-21 06:02:36,065 [myid:] - INFO [Thread-21:LeaderElection@130] - 14 -> 1
2016-07-21 06:02:36,065 [myid:] - INFO [Thread-21:LeaderElection@130] - 22 -> 1
2016-07-21 06:02:36,065 [myid:] - INFO [Thread-21:LeaderElection@130] - 21 -> 1
2016-07-21 06:02:36,065 [myid:] - INFO [Thread-21:LeaderElection@130] - 13 -> 1
2016-07-21 06:02:36,065 [myid:] - INFO [Thread-21:LeaderElection@130] - 24 -> 1
2016-07-21 06:02:36,066 [myid:] - INFO [Thread-21:LeaderElection@130] - 16 -> 1
2016-07-21 06:02:36,057 [myid:] - INFO [Thread-19:LeaderElection@130] - 18 -> 1
2016-07-21 06:02:36,056 [myid:] - INFO [Thread-12:LeaderElection@130] - 22 -> 1
2016-07-21 06:02:36,066 [myid:] - INFO [Thread-19:LeaderElection@130] - 17 -> 1
2016-07-21 06:02:36,066 [myid:] - INFO [Thread-19:LeaderElection@130] - 25 -> 1
2016-07-21 06:02:36,066 [myid:] - INFO [Thread-19:LeaderElection@130] - 20 -> 1
2016-07-21 06:02:36,066 [myid:] - INFO [Thread-19:LeaderElection@130] - 19 -> 1
2016-07-21 06:02:36,064 [myid:] - INFO [Thread-25:LeaderElection@130] - 13 -> 1
2016-07-21 06:02:36,066 [myid:] - INFO [Thread-25:LeaderElection@130] - 21 -> 1
2016-07-21 06:02:36,066 [myid:] - INFO [Thread-25:LeaderElection@130] - 16 -> 1
2016-07-21 06:02:36,067 [myid:] - INFO [Thread-25:LeaderElection@130] - 24 -> 1
2016-07-21 06:02:36,064 [myid:] - INFO [Thread-26:LeaderElection@130] - 16 -> 1
2016-07-21 06:02:36,064 [myid:] - INFO [Thread-27:LeaderElection@187] - Server address: /127.0.0.1:11269
2016-07-21 06:02:36,064 [myid:] - INFO [Thread-30:LeaderElection@187] - Server address: /127.0.0.1:11249
2016-07-21 06:02:36,064 [myid:] - INFO [Thread-28:LeaderElection@187] - Server address: /127.0.0.1:11249
2016-07-21 06:02:36,064 [myid:] - INFO [Thread-6:LeaderElection@187] - Server address: /127.0.0.1:11253
2016-07-21 06:02:36,068 [myid:] - INFO [Thread-28:LeaderElection@187] - Server address: /127.0.0.1:11251
2016-07-21 06:02:36,064 [myid:] - INFO [Thread-29:LeaderElection@187] - Server address: /127.0.0.1:11257
2016-07-21 06:02:36,068 [myid:] - INFO [Thread-28:LeaderElection@187] - Server address: /127.0.0.1:11255
2016-07-21 06:02:36,068 [myid:] - INFO [Thread-29:LeaderElection@187] - Server address: /127.0.0.1:11263
2016-07-21 06:02:36,063 [myid:] - INFO [Thread-8:LeaderElection@187] - Server address: /127.0.0.1:11247
2016-07-21 06:02:36,063 [myid:] - INFO [Thread-11:LeaderElection@187] - Server address: /127.0.0.1:11263
2016-07-21 06:02:36,063 [myid:] - INFO [Thread-5:LeaderElection@187] - Server address: /127.0.0.1:11255
2016-07-21 06:02:36,063 [myid:] - INFO [Thread-4:LeaderElection@187] - Server address: /127.0.0.1:11265
2016-07-21 06:02:36,063 [myid:] - INFO [Thread-1:LeaderElection@187] - Server address: /127.0.0.1:11251
2016-07-21 06:02:40,547 [myid:] - INFO [Thread-4:LeaderElection@187] - Server address: /127.0.0.1:11271
2016-07-21 06:02:36,063 [myid:] - INFO [Thread-17:LeaderElection@187] - Server address: /127.0.0.1:11245
2016-07-21 06:02:36,062 [myid:] - INFO [Thread-0:LeaderElection@187] - Server address: /127.0.0.1:11261
2016-07-21 06:02:40,547 [myid:] - INFO [Thread-17:LeaderElection@187] - Server address: /127.0.0.1:11247
2016-07-21 06:02:40,548 [myid:] - INFO [Thread-0:LeaderElection@187] - Server address: /127.0.0.1:11267
2016-07-21 06:02:36,062 [myid:] - INFO [Thread-7:LeaderElection@187] - Server address: /127.0.0.1:11255
2016-07-21 06:02:36,062 [myid:] - INFO [Thread-15:LeaderElection@187] - Server address: /127.0.0.1:11235
2016-07-21 06:02:36,062 [myid:] - INFO [Thread-10:LeaderElection@187] - Server address: /127.0.0.1:11235
2016-07-21 06:02:40,548 [myid:] - INFO [Thread-15:LeaderElection@187] - Server address: /127.0.0.1:11237
2016-07-21 06:02:36,062 [myid:] - INFO [Thread-9:LeaderElection@187] - Server address: /127.0.0.1:11235
2016-07-21 06:02:36,061 [myid:] - INFO [Thread-13:LeaderElection@187] - Server address: /127.0.0.1:11245
2016-07-21 06:02:36,061 [myid:] - INFO [Thread-3:LeaderElection@187] - Server address: /127.0.0.1:11235
2016-07-21 06:02:40,549 [myid:] - INFO [Thread-13:LeaderElection@187] - Server address: /127.0.0.1:11247
2016-07-21 06:02:40,549 [myid:] - INFO [Thread-3:LeaderElection@187] - Server address: /127.0.0.1:11237
2016-07-21 06:02:36,061 [myid:] - INFO [Thread-22:LeaderElection@130] - 7 -> 1
2016-07-21 06:02:40,549 [myid:] - INFO [Thread-3:LeaderElection@187] - Server address: /127.0.0.1:11239
2016-07-21 06:02:40,550 [myid:] - INFO [Thread-3:LeaderElection@187] - Server address: /127.0.0.1:11241
2016-07-21 06:02:40,549 [myid:] - INFO [Thread-13:LeaderElection@187] - Server address: /127.0.0.1:11249
2016-07-21 06:02:40,550 [myid:] - INFO [Thread-3:LeaderElection@187] - Server address: /127.0.0.1:11243
2016-07-21 06:02:40,550 [myid:] - INFO [Thread-13:LeaderElection@187] - Server address: /127.0.0.1:11251
2016-07-21 06:02:40,550 [myid:] - INFO [Thread-3:LeaderElection@187] - Server address: /127.0.0.1:11245
2016-07-21 06:02:40,550 [myid:] - INFO [Thread-13:LeaderElection@187] - Server address: /127.0.0.1:11255
2016-07-21 06:02:40,550 [myid:] - INFO [Thread-3:LeaderElection@187] - Server address: /127.0.0.1:11247
2016-07-21 06:02:40,550 [myid:] - INFO [Thread-13:LeaderElection@187] - Server address: /127.0.0.1:11253
2016-07-21 06:02:40,550 [myid:] - INFO [Thread-3:LeaderElection@187] - Server address: /127.0.0.1:11249
2016-07-21 06:02:40,551 [myid:] - INFO [Thread-13:LeaderElection@187] - Server address: /127.0.0.1:11259
2016-07-21 06:02:40,551 [myid:] - INFO [Thread-3:LeaderElection@187] - Server address: /127.0.0.1:11251
2016-07-21 06:02:40,549 [myid:] - INFO [Thread-9:LeaderElection@187] - Server address: /127.0.0.1:11237
2016-07-21 06:02:40,551 [myid:] - INFO [Thread-3:LeaderElection@187] - Server address: /127.0.0.1:11255
2016-07-21 06:02:40,551 [myid:] - INFO [Thread-9:LeaderElection@187] - Server address: /127.0.0.1:11239
2016-07-21 06:02:40,549 [myid:] - INFO [Thread-15:LeaderElection@187] - Server address: /127.0.0.1:11239
2016-07-21 06:02:40,548 [myid:] - INFO [Thread-10:LeaderElection@187] - Server address: /127.0.0.1:11237
2016-07-21 06:02:40,551 [myid:] - INFO [Thread-15:LeaderElection@187] - Server address: /127.0.0.1:11241
2016-07-21 06:02:40,548 [myid:] - INFO [Thread-7:LeaderElection@187] - Server address: /127.0.0.1:11253
2016-07-21 06:02:40,552 [myid:] - INFO [Thread-15:LeaderElection@187] - Server address: /127.0.0.1:11243
2016-07-21 06:02:40,548 [myid:] - INFO [Thread-0:LeaderElection@187] - Server address: /127.0.0.1:11265
2016-07-21 06:02:40,548 [myid:] - INFO [Thread-17:LeaderElection@187] - Server address: /127.0.0.1:11249
2016-07-21 06:02:40,552 [myid:] - INFO [Thread-0:LeaderElection@187] - Server address: /127.0.0.1:11271
2016-07-21 06:02:40,552 [myid:] - INFO [Thread-17:LeaderElection@187] - Server address: /127.0.0.1:11251
2016-07-21 06:02:40,552 [myid:] - INFO [Thread-17:LeaderElection@187] - Server address: /127.0.0.1:11255
2016-07-21 06:02:40,547 [myid:] - INFO [Thread-4:LeaderElection@187] - Server address: /127.0.0.1:11269
2016-07-21 06:02:40,547 [myid:] - INFO [Thread-1:LeaderElection@187] - Server address: /127.0.0.1:11255
2016-07-21 06:02:40,547 [myid:] - INFO [Thread-5:LeaderElection@187] - Server address: /127.0.0.1:11253
2016-07-21 06:02:37,067 [myid:] - INFO [Thread-26:LeaderElection@187] - Server address: /127.0.0.1:11221
2016-07-21 06:02:37,067 [myid:] - INFO [Thread-25:LeaderElection@187] - Server address: /127.0.0.1:11221
2016-07-21 06:02:37,066 [myid:] - INFO [Thread-21:LeaderElection@187] - Server address: /127.0.0.1:11221
2016-07-21 06:02:37,061 [myid:] - INFO [Thread-14:LeaderElection@187] - Server address: /127.0.0.1:11221
2016-07-21 06:02:37,053 [myid:] - INFO [Thread-18:LeaderElection@187] - Server address: /127.0.0.1:11221
2016-07-21 06:02:40,554 [myid:] - INFO [Thread-14:LeaderElection@187] - Server address: /127.0.0.1:11223
2016-07-21 06:02:40,554 [myid:] - INFO [Thread-18:LeaderElection@187] - Server address: /127.0.0.1:11223
2016-07-21 06:02:40,554 [myid:] - INFO [Thread-14:LeaderElection@187] - Server address: /127.0.0.1:11225
2016-07-21 06:02:40,554 [myid:] - INFO [Thread-18:LeaderElection@187] - Server address: /127.0.0.1:11225
2016-07-21 06:02:40,554 [myid:] - INFO [Thread-18:LeaderElection@187] - Server address: /127.0.0.1:11227
2016-07-21 06:02:40,555 [myid:] - INFO [Thread-18:LeaderElection@187] - Server address: /127.0.0.1:11229
2016-07-21 06:02:37,051 [myid:] - INFO [Thread-24:LeaderElection@187] - Server address: /127.0.0.1:11221
2016-07-21 06:02:37,013 [myid:] - INFO [Thread-23:LeaderElection@187] - Server address: /127.0.0.1:11221
2016-07-21 06:02:37,012 [myid:] - INFO [Thread-20:LeaderElection@187] - Server address: /127.0.0.1:11221
2016-07-21 06:02:40,555 [myid:] - INFO [Thread-23:LeaderElection@187] - Server address: /127.0.0.1:11223
2016-07-21 06:02:37,012 [myid:] - INFO [Thread-16:LeaderElection@187] - Server address: /127.0.0.1:11221
2016-07-21 06:02:36,069 [myid:] - INFO [Thread-11:LeaderElection@187] - Server address: /127.0.0.1:11261
2016-07-21 06:02:40,556 [myid:] - INFO [Thread-16:LeaderElection@187] - Server address: /127.0.0.1:11223
2016-07-21 06:02:40,556 [myid:] - INFO [Thread-11:LeaderElection@187] - Server address: /127.0.0.1:11267
2016-07-21 06:02:40,556 [myid:] - INFO [Thread-16:LeaderElection@187] - Server address: /127.0.0.1:11225
2016-07-21 06:02:40,556 [myid:] - INFO [Thread-11:LeaderElection@187] - Server address: /127.0.0.1:11265
2016-07-21 06:02:40,556 [myid:] - INFO [Thread-16:LeaderElection@187] - Server address: /127.0.0.1:11227
2016-07-21 06:02:40,556 [myid:] - INFO [Thread-11:LeaderElection@187] - Server address: /127.0.0.1:11271
2016-07-21 06:02:40,556 [myid:] - INFO [Thread-16:LeaderElection@187] - Server address: /127.0.0.1:11229
2016-07-21 06:02:40,557 [myid:] - INFO [Thread-11:LeaderElection@187] - Server address: /127.0.0.1:11269
2016-07-21 06:02:40,557 [myid:] - INFO [Thread-16:LeaderElection@187] - Server address: /127.0.0.1:11231
2016-07-21 06:02:40,557 [myid:] - INFO [Thread-11:LeaderElection@187] - Server address: /127.0.0.1:11275
2016-07-21 06:02:40,557 [myid:] - INFO [Thread-16:LeaderElection@187] - Server address: /127.0.0.1:11233
2016-07-21 06:02:40,557 [myid:] - INFO [Thread-11:LeaderElection@187] - Server address: /127.0.0.1:11273
2016-07-21 06:02:36,069 [myid:] - INFO [Thread-8:LeaderElection@187] - Server address: /127.0.0.1:11249
2016-07-21 06:02:36,069 [myid:] - INFO [Thread-29:LeaderElection@187] - Server address: /127.0.0.1:11261
2016-07-21 06:02:36,068 [myid:] - INFO [Thread-28:LeaderElection@187] - Server address: /127.0.0.1:11253
2016-07-21 06:02:36,068 [myid:] - INFO [Thread-6:LeaderElection@187] - Server address: /127.0.0.1:11259
2016-07-21 06:02:36,068 [myid:] - INFO [Thread-30:LeaderElection@187] - Server address: /127.0.0.1:11251
2016-07-21 06:02:45,396 [myid:] - INFO [Thread-6:LeaderElection@187] - Server address: /127.0.0.1:11257
2016-07-21 06:02:36,068 [myid:] - INFO [Thread-27:LeaderElection@187] - Server address: /127.0.0.1:11275
2016-07-21 06:02:36,066 [myid:] - INFO [Thread-19:LeaderElection@130] - 14 -> 1
2016-07-21 06:02:45,397 [myid:] - INFO [Thread-19:LeaderElection@130] - 22 -> 1
2016-07-21 06:02:45,397 [myid:] - INFO [Thread-19:LeaderElection@130] - 13 -> 1
2016-07-21 06:02:45,397 [myid:] - INFO [Thread-19:LeaderElection@130] - 24 -> 1
2016-07-21 06:02:36,066 [myid:] - INFO [Thread-12:LeaderElection@130] - 13 -> 1
2016-07-21 06:02:45,397 [myid:] - INFO [Thread-12:LeaderElection@130] - 21 -> 1
2016-07-21 06:02:45,397 [myid:] - INFO [Thread-12:LeaderElection@130] - 24 -> 1
2016-07-21 06:02:45,398 [myid:] - INFO [Thread-12:LeaderElection@130] - 16 -> 1
2016-07-21 06:02:45,398 [myid:] - INFO [Thread-12:LeaderElection@130] - 23 -> 1
2016-07-21 06:02:45,397 [myid:] - INFO [Thread-19:LeaderElection@130] - 16 -> 1
2016-07-21 06:02:45,398 [myid:] - INFO [Thread-19:LeaderElection@130] - 23 -> 1
2016-07-21 06:02:45,397 [myid:] - INFO [Thread-27:LeaderElection@187] - Server address: /127.0.0.1:11273
2016-07-21 06:02:45,396 [myid:] - INFO [Thread-6:LeaderElection@187] - Server address: /127.0.0.1:11263
2016-07-21 06:02:45,396 [myid:] - INFO [Thread-30:LeaderElection@187] - Server address: /127.0.0.1:11255
2016-07-21 06:02:45,399 [myid:] - INFO [Thread-6:LeaderElection@187] - Server address: /127.0.0.1:11261
2016-07-21 06:02:45,396 [myid:] - INFO [Thread-28:LeaderElection@187] - Server address: /127.0.0.1:11259
2016-07-21 06:02:45,399 [myid:] - INFO [Thread-6:LeaderElection@187] - Server address: /127.0.0.1:11267
2016-07-21 06:02:45,399 [myid:] - INFO [Thread-28:LeaderElection@187] - Server address: /127.0.0.1:11257
2016-07-21 06:02:44,735 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@74] - TEST METHOD FAILED testLE
java.lang.AssertionError: Threads didn't join
at org.junit.Assert.fail(Assert.java:91)
at org.apache.zookeeper.test.LETest.testLE(LETest.java:120)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:55)
at org.junit.rules.TestWatchman$1.evaluate(TestWatchman.java:48)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:76)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184)
at org.junit.runners.ParentRunner.run(ParentRunner.java:236)
at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:39)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:532)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1179)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:1030)
2016-07-21 06:02:40,558 [myid:] - INFO [Thread-29:LeaderElection@187] - Server address: /127.0.0.1:11267
2016-07-21 06:02:40,558 [myid:] - INFO [Thread-8:LeaderElection@187] - Server address: /127.0.0.1:11251
2016-07-21 06:02:40,557 [myid:] - INFO [Thread-11:LeaderElection@187] - Server address: /127.0.0.1:11279
2016-07-21 06:02:40,557 [myid:] - INFO [Thread-16:LeaderElection@187] - Server address: /127.0.0.1:11235
2016-07-21 06:02:45,401 [myid:] - INFO [Thread-11:LeaderElection@187] - Server address: /127.0.0.1:11277
2016-07-21 06:02:40,555 [myid:] - INFO [Thread-23:LeaderElection@187] - Server address: /127.0.0.1:11225
2016-07-21 06:02:40,555 [myid:] - INFO [Thread-20:LeaderElection@187] - Server address: /127.0.0.1:11223
2016-07-21 06:02:40,555 [myid:] - INFO [Thread-24:LeaderElection@187] - Server address: /127.0.0.1:11223
2016-07-21 06:02:40,555 [myid:] - INFO [Thread-14:LeaderElection@187] - Server address: /127.0.0.1:11227
2016-07-21 06:02:45,402 [myid:] - INFO [Thread-24:LeaderElection@187] - Server address: /127.0.0.1:11225
2016-07-21 06:02:40,555 [myid:] - INFO [Thread-18:LeaderElection@187] - Server address: /127.0.0.1:11231
2016-07-21 06:02:40,554 [myid:] - INFO [Thread-21:LeaderElection@187] - Server address: /127.0.0.1:11223
2016-07-21 06:02:40,554 [myid:] - INFO [Thread-25:LeaderElection@187] - Server address: /127.0.0.1:11223
2016-07-21 06:02:45,403 [myid:] - INFO [Thread-21:LeaderElection@187] - Server address: /127.0.0.1:11225
2016-07-21 06:02:40,553 [myid:] - INFO [Thread-26:LeaderElection@187] - Server address: /127.0.0.1:11223
2016-07-21 06:02:40,553 [myid:] - INFO [Thread-5:LeaderElection@187] - Server address: /127.0.0.1:11259
2016-07-21 06:02:45,403 [myid:] - INFO [Thread-26:LeaderElection@187] - Server address: /127.0.0.1:11225
2016-07-21 06:02:45,403 [myid:] - INFO [Thread-5:LeaderElection@187] - Server address: /127.0.0.1:11257
2016-07-21 06:02:40,553 [myid:] - INFO [Thread-1:LeaderElection@187] - Server address: /127.0.0.1:11253
2016-07-21 06:02:40,553 [myid:] - INFO [Thread-4:LeaderElection@187] - Server address: /127.0.0.1:11275
2016-07-21 06:02:45,404 [myid:] - INFO [Thread-1:LeaderElection@187] - Server address: /127.0.0.1:11259
2016-07-21 06:02:40,553 [myid:] - INFO [Thread-17:LeaderElection@187] - Server address: /127.0.0.1:11253
2016-07-21 06:02:45,404 [myid:] - INFO [Thread-1:LeaderElection@187] - Server address: /127.0.0.1:11257
2016-07-21 06:02:40,552 [myid:] - INFO [Thread-0:LeaderElection@187] - Server address: /127.0.0.1:11269
2016-07-21 06:02:40,552 [myid:] - INFO [Thread-15:LeaderElection@187] - Server address: /127.0.0.1:11245
2016-07-21 06:02:45,405 [myid:] - INFO [Thread-0:LeaderElection@187] - Server address: /127.0.0.1:11275
2016-07-21 06:02:40,552 [myid:] - INFO [Thread-3:LeaderElection@187] - Server address: /127.0.0.1:11253
2016-07-21 06:02:40,552 [myid:] - INFO [Thread-7:LeaderElection@187] - Server address: /127.0.0.1:11259
2016-07-21 06:02:40,551 [myid:] - INFO [Thread-10:LeaderElection@187] - Server address: /127.0.0.1:11239
2016-07-21 06:02:40,551 [myid:] - INFO [Thread-9:LeaderElection@187] - Server address: /127.0.0.1:11241
2016-07-21 06:02:40,551 [myid:] - INFO [Thread-13:LeaderElection@187] - Server address: /127.0.0.1:11257
2016-07-21 06:02:45,406 [myid:] - INFO [Thread-9:LeaderElection@187] - Server address: /127.0.0.1:11243
2016-07-21 06:02:45,406 [myid:] - INFO [Thread-13:LeaderElection@187] - Server address: /127.0.0.1:11263
2016-07-21 06:02:45,406 [myid:] - INFO [Thread-9:LeaderElection@187] - Server address: /127.0.0.1:11245
2016-07-21 06:02:45,406 [myid:] - INFO [Thread-13:LeaderElection@187] - Server address: /127.0.0.1:11261
2016-07-21 06:02:40,549 [myid:] - INFO [Thread-22:LeaderElection@130] - 18 -> 1
2016-07-21 06:02:45,407 [myid:] - INFO [Thread-22:LeaderElection@130] - 17 -> 1
2016-07-21 06:02:45,407 [myid:] - INFO [Thread-22:LeaderElection@130] - 20 -> 1
2016-07-21 06:02:45,407 [myid:] - INFO [Thread-22:LeaderElection@130] - 19 -> 1
2016-07-21 06:02:45,407 [myid:] - INFO [Thread-22:LeaderElection@130] - 14 -> 1
2016-07-21 06:02:45,407 [myid:] - INFO [Thread-22:LeaderElection@130] - 22 -> 1
2016-07-21 06:02:45,407 [myid:] - INFO [Thread-22:LeaderElection@130] - 13 -> 1
2016-07-21 06:02:45,407 [myid:] - INFO [Thread-22:LeaderElection@130] - 21 -> 1
2016-07-21 06:02:45,407 [myid:] - INFO [Thread-22:LeaderElection@130] - 24 -> 1
2016-07-21 06:02:45,407 [myid:] - INFO [Thread-22:LeaderElection@130] - 16 -> 1
2016-07-21 06:02:45,408 [myid:] - INFO [Thread-22:LeaderElection@130] - 23 -> 1
2016-07-21 06:02:45,406 [myid:] - INFO [Thread-13:LeaderElection@187] - Server address: /127.0.0.1:11267
2016-07-21 06:02:45,406 [myid:] - INFO [Thread-9:LeaderElection@187] - Server address: /127.0.0.1:11247
2016-07-21 06:02:45,406 [myid:] - INFO [Thread-10:LeaderElection@187] - Server address: /127.0.0.1:11241
2016-07-21 06:02:45,405 [myid:] - INFO [Thread-7:LeaderElection@187] - Server address: /127.0.0.1:11257
2016-07-21 06:02:45,405 [myid:] - INFO [Thread-3:LeaderElection@187] - Server address: /127.0.0.1:11259
2016-07-21 06:02:45,408 [myid:] - INFO [Thread-10:LeaderElection@187] - Server address: /127.0.0.1:11243
2016-07-21 06:02:45,409 [myid:] - INFO [Thread-3:LeaderElection@187] - Server address: /127.0.0.1:11257
2016-07-21 06:02:45,409 [myid:] - INFO [Thread-10:LeaderElection@187] - Server address: /127.0.0.1:11245
2016-07-21 06:02:45,409 [myid:] - INFO [Thread-3:LeaderElection@187] - Server address: /127.0.0.1:11263
2016-07-21 06:02:45,405 [myid:] - INFO [Thread-0:LeaderElection@187] - Server address: /127.0.0.1:11273
2016-07-21 06:02:45,409 [myid:] - INFO [Thread-3:LeaderElection@187] - Server address: /127.0.0.1:11261
2016-07-21 06:02:45,405 [myid:] - INFO [Thread-15:LeaderElection@187] - Server address: /127.0.0.1:11247
2016-07-21 06:02:45,404 [myid:] - INFO [Thread-1:LeaderElection@187] - Server address: /127.0.0.1:11263
2016-07-21 06:02:45,409 [myid:] - INFO [Thread-15:LeaderElection@187] - Server address: /127.0.0.1:11249
2016-07-21 06:02:45,410 [myid:] - INFO [Thread-1:LeaderElection@187] - Server address: /127.0.0.1:11261
2016-07-21 06:02:45,404 [myid:] - INFO [Thread-17:LeaderElection@187] - Server address: /127.0.0.1:11259
2016-07-21 06:02:45,410 [myid:] - INFO [Thread-1:LeaderElection@187] - Server address: /127.0.0.1:11267
2016-07-21 06:02:45,410 [myid:] - INFO [Thread-17:LeaderElection@187] - Server address: /127.0.0.1:11257
2016-07-21 06:02:45,404 [myid:] - INFO [Thread-4:LeaderElection@187] - Server address: /127.0.0.1:11273
2016-07-21 06:02:45,404 [myid:] - INFO [Thread-5:LeaderElection@187] - Server address: /127.0.0.1:11263
2016-07-21 06:02:45,403 [myid:] - INFO [Thread-26:LeaderElection@187] - Server address: /127.0.0.1:11227
2016-07-21 06:02:45,403 [myid:] - INFO [Thread-21:LeaderElection@187] - Server address: /127.0.0.1:11227
2016-07-21 06:02:45,411 [myid:] - INFO [Thread-26:LeaderElection@187] - Server address: /127.0.0.1:11229
2016-07-21 06:02:45,403 [myid:] - INFO [Thread-25:LeaderElection@187] - Server address: /127.0.0.1:11225
2016-07-21 06:02:45,402 [myid:] - INFO [Thread-18:LeaderElection@187] - Server address: /127.0.0.1:11233
2016-07-21 06:02:45,402 [myid:] - INFO [Thread-24:LeaderElection@187] - Server address: /127.0.0.1:11227
2016-07-21 06:02:45,402 [myid:] - INFO [Thread-14:LeaderElection@187] - Server address: /127.0.0.1:11229
2016-07-21 06:02:45,401 [myid:] - INFO [Thread-20:LeaderElection@187] - Server address: /127.0.0.1:11225
2016-07-21 06:02:45,412 [myid:] - INFO [Thread-14:LeaderElection@187] - Server address: /127.0.0.1:11231
2016-07-21 06:02:45,401 [myid:] - INFO [Thread-11:LeaderElection@124] - Election tally:
2016-07-21 06:02:45,412 [myid:] - INFO [Thread-14:LeaderElection@187] - Server address: /127.0.0.1:11233
2016-07-21 06:02:45,401 [myid:] - INFO [Thread-23:LeaderElection@187] - Server address: /127.0.0.1:11227
2016-07-21 06:02:45,401 [myid:] - INFO [Thread-16:LeaderElection@187] - Server address: /127.0.0.1:11237
2016-07-21 06:02:45,401 [myid:] - INFO [Thread-8:LeaderElection@187] - Server address: /127.0.0.1:11255
2016-07-21 06:02:45,400 [myid:] - INFO [Thread-29:LeaderElection@187] - Server address: /127.0.0.1:11265
2016-07-21 06:02:45,400 [myid:] - INFO [main:ZKTestCase$1@65] - FAILED testLE
java.lang.AssertionError: Threads didn't join
at org.junit.Assert.fail(Assert.java:91)
at org.apache.zookeeper.test.LETest.testLE(LETest.java:120)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:55)
at org.junit.rules.TestWatchman$1.evaluate(TestWatchman.java:48)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:76)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184)
at org.junit.runners.ParentRunner.run(ParentRunner.java:236)
at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:39)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:532)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1179)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:1030)
2016-07-21 06:02:45,399 [myid:] - INFO [Thread-28:LeaderElection@187] - Server address: /127.0.0.1:11263
2016-07-21 06:02:45,399 [myid:] - INFO [Thread-6:LeaderElection@187] - Server address: /127.0.0.1:11265
2016-07-21 06:02:45,399 [myid:] - INFO [Thread-30:LeaderElection@187] - Server address: /127.0.0.1:11253
2016-07-21 06:02:45,399 [myid:] - INFO [Thread-27:LeaderElection@187] - Server address: /127.0.0.1:11279
2016-07-21 06:02:45,418 [myid:] - INFO [Thread-30:LeaderElection@187] - Server address: /127.0.0.1:11259
2016-07-21 06:02:45,418 [myid:] - INFO [Thread-6:LeaderElection@187] - Server address: /127.0.0.1:11271
2016-07-21 06:02:45,417 [myid:] - INFO [Thread-28:LeaderElection@187] - Server address: /127.0.0.1:11261
2016-07-21 06:02:45,417 [myid:] - INFO [main:ZKTestCase$1@55] - FINISHED testLE
2016-07-21 06:02:45,413 [myid:] - INFO [Thread-29:LeaderElection@187] - Server address: /127.0.0.1:11271
2016-07-21 06:02:45,413 [myid:] - INFO [Thread-8:LeaderElection@187] - Server address: /127.0.0.1:11253
2016-07-21 06:02:45,413 [myid:] - INFO [Thread-16:LeaderElection@187] - Server address: /127.0.0.1:11239
2016-07-21 06:02:45,413 [myid:] - INFO [Thread-23:LeaderElection@187] - Server address: /127.0.0.1:11229
2016-07-21 06:02:45,413 [myid:] - INFO [Thread-14:LeaderElection@187] - Server address: /127.0.0.1:11235
2016-07-21 06:02:45,412 [myid:] - INFO [Thread-11:LeaderElection@130] - 15 -> 1
2016-07-21 06:02:45,420 [myid:] - INFO [Thread-11:LeaderElection@130] - 13 -> 1
2016-07-21 06:02:45,420 [myid:] - INFO [Thread-11:LeaderElection@130] - 21 -> 1
2016-07-21 06:02:45,420 [myid:] - INFO [Thread-11:LeaderElection@130] - 27 -> 1
2016-07-21 06:02:45,420 [myid:] - INFO [Thread-11:LeaderElection@130] - 17 -> 1
2016-07-21 06:02:45,420 [myid:] - INFO [Thread-11:LeaderElection@130] - 18 -> 1
2016-07-21 06:02:45,420 [myid:] - INFO [Thread-11:LeaderElection@130] - 11 -> 1
2016-07-21 06:02:45,420 [myid:] - INFO [Thread-11:LeaderElection@130] - 29 -> 23
2016-07-21 06:02:45,412 [myid:] - INFO [Thread-20:LeaderElection@187] - Server address: /127.0.0.1:11227
2016-07-21 06:02:45,412 [myid:] - INFO [Thread-24:LeaderElection@187] - Server address: /127.0.0.1:11229
2016-07-21 06:02:45,412 [myid:] - INFO [Thread-18:LeaderElection@187] - Server address: /127.0.0.1:11235
2016-07-21 06:02:45,411 [myid:] - INFO [Thread-25:LeaderElection@187] - Server address: /127.0.0.1:11227
2016-07-21 06:02:45,411 [myid:] - INFO [Thread-26:LeaderElection@187] - Server address: /127.0.0.1:11231
2016-07-21 06:02:45,421 [myid:] - INFO [Thread-25:LeaderElection@187] - Server address: /127.0.0.1:11229
2016-07-21 06:02:45,411 [myid:] - INFO [Thread-21:LeaderElection@187] - Server address: /127.0.0.1:11229
2016-07-21 06:02:45,411 [myid:] - INFO [Thread-5:LeaderElection@187] - Server address: /127.0.0.1:11261
2016-07-21 06:02:45,422 [myid:] - INFO [Thread-21:LeaderElection@187] - Server address: /127.0.0.1:11231
2016-07-21 06:02:45,422 [myid:] - INFO [Thread-5:LeaderElection@187] - Server address: /127.0.0.1:11267
2016-07-21 06:02:45,411 [myid:] - INFO [Thread-4:LeaderElection@187] - Server address: /127.0.0.1:11279
2016-07-21 06:02:45,410 [myid:] - INFO [Thread-17:LeaderElection@187] - Server address: /127.0.0.1:11263
2016-07-21 06:02:45,422 [myid:] - INFO [Thread-4:LeaderElection@187] - Server address: /127.0.0.1:11277
2016-07-21 06:02:45,410 [myid:] - INFO [Thread-1:LeaderElection@187] - Server address: /127.0.0.1:11265
2016-07-21 06:02:45,410 [myid:] - INFO [Thread-15:LeaderElection@187] - Server address: /127.0.0.1:11251
2016-07-21 06:02:45,409 [myid:] - INFO [Thread-3:LeaderElection@187] - Server address: /127.0.0.1:11267
2016-07-21 06:02:45,409 [myid:] - INFO [Thread-0:LeaderElection@187] - Server address: /127.0.0.1:11279
2016-07-21 06:02:45,409 [myid:] - INFO [Thread-10:LeaderElection@187] - Server address: /127.0.0.1:11247
2016-07-21 06:02:45,408 [myid:] - INFO [Thread-7:LeaderElection@187] - Server address: /127.0.0.1:11263
2016-07-21 06:02:45,408 [myid:] - INFO [Thread-9:LeaderElection@187] - Server address: /127.0.0.1:11249
2016-07-21 06:02:45,408 [myid:] - INFO [Thread-13:LeaderElection@187] - Server address: /127.0.0.1:11265
2016-07-21 06:02:45,424 [myid:] - INFO [Thread-7:LeaderElection@187] - Server address: /127.0.0.1:11261
2016-07-21 06:02:45,424 [myid:] - INFO [Thread-13:LeaderElection@187] - Server address: /127.0.0.1:11271
2016-07-21 06:02:45,424 [myid:] - INFO [Thread-10:LeaderElection@187] - Server address: /127.0.0.1:11249
2016-07-21 06:02:45,423 [myid:] - INFO [Thread-0:LeaderElection@187] - Server address: /127.0.0.1:11277
2016-07-21 06:02:45,425 [myid:] - INFO [Thread-10:LeaderElection@187] - Server address: /127.0.0.1:11251
{noformat}
flaky, flaky-test 9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 34 weeks, 1 day ago 0|i31cjr:
ZooKeeper ZOOKEEPER-2482

Flaky Test: org.apache.zookeeper.test.ClientPortBindTest.testBindByAddress

Test Closed Major Fixed Michael Han Michael Han Michael Han 21/Jul/16 17:40   17/May/17 23:43 15/Aug/16 18:46 3.5.2 3.5.3 server, tests   0 1   From https://builds.apache.org/job/ZooKeeper_branch34/1587/

{noformat}
Error Message

No such device
Stacktrace

java.net.SocketException: No such device
at java.net.NetworkInterface.isLoopback0(Native Method)
at java.net.NetworkInterface.isLoopback(NetworkInterface.java:339)
at org.apache.zookeeper.test.ClientPortBindTest.testBindByAddress(ClientPortBindTest.java:61)
at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:55)
Standard Output

2016-07-21 05:58:50,388 [myid:] - INFO [main:ZKTestCase$1@50] - STARTING testBindByAddress
2016-07-21 05:58:50,393 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@53] - RUNNING TEST METHOD testBindByAddress
2016-07-21 05:58:50,405 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@74] - TEST METHOD FAILED testBindByAddress
java.net.SocketException: No such device
at java.net.NetworkInterface.isLoopback0(Native Method)
at java.net.NetworkInterface.isLoopback(NetworkInterface.java:339)
at org.apache.zookeeper.test.ClientPortBindTest.testBindByAddress(ClientPortBindTest.java:61)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:55)
at org.junit.rules.TestWatchman$1.evaluate(TestWatchman.java:48)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:76)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184)
at org.junit.runners.ParentRunner.run(ParentRunner.java:236)
at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:39)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:532)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1179)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:1030)
2016-07-21 05:58:50,408 [myid:] - INFO [main:ZKTestCase$1@65] - FAILED testBindByAddress
java.net.SocketException: No such device
at java.net.NetworkInterface.isLoopback0(Native Method)
at java.net.NetworkInterface.isLoopback(NetworkInterface.java:339)
at org.apache.zookeeper.test.ClientPortBindTest.testBindByAddress(ClientPortBindTest.java:61)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:55)
at org.junit.rules.TestWatchman$1.evaluate(TestWatchman.java:48)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:76)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184)
at org.junit.runners.ParentRunner.run(ParentRunner.java:236)
at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:39)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:532)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1179)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:1030)
2016-07-21 05:58:50,409 [myid:] - INFO [main:ZKTestCase$1@55] - FINISHED testBindByAddress
{noformat}
flaky, flaky-test 9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 31 weeks, 3 days ago 0|i31cjb:
ZooKeeper ZOOKEEPER-2481

ZOOKEEPER-3170 Flaky Test: testZeroWeightQuorum

Sub-task Closed Major Cannot Reproduce Andor Molnar Michael Han Michael Han 21/Jul/16 17:33   19/Dec/19 18:02 25/Oct/18 11:11 3.4.8, 3.5.2 3.5.5 server, tests   0 4   See https://builds.apache.org/job/ZooKeeper-trunk-openjdk7/1098/

{noformat}
Error Message

Threads didn't join
Stacktrace

junit.framework.AssertionFailedError: Threads didn't join
at org.apache.zookeeper.test.FLEZeroWeightTest.testZeroWeightQuorum(FLEZeroWeightTest.java:167)
at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:79)
Standard Output

2016-07-21 04:24:14,065 [myid:] - INFO [main:JUnit4ZKTestRunner@47] - No test.method specified. using default methods.
2016-07-21 04:24:14,158 [myid:] - INFO [main:JUnit4ZKTestRunner@47] - No test.method specified. using default methods.
2016-07-21 04:24:14,176 [myid:] - INFO [main:ZKTestCase$1@55] - STARTING testZeroWeightQuorum
2016-07-21 04:24:14,180 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@77] - RUNNING TEST METHOD testZeroWeightQuorum
2016-07-21 04:24:14,180 [myid:] - INFO [main:FLEZeroWeightTest@143] - TestZeroWeightQuorum: testZeroWeightQuorum, 9
2016-07-21 04:24:14,183 [myid:] - INFO [main:PortAssignment@157] - Single test process using ports from 11221 - 32767.
2016-07-21 04:24:14,187 [myid:] - INFO [main:PortAssignment@85] - Assigned port 11222 from range 11221 - 32767.
2016-07-21 04:24:14,189 [myid:] - INFO [main:PortAssignment@85] - Assigned port 11223 from range 11221 - 32767.
2016-07-21 04:24:14,189 [myid:] - INFO [main:PortAssignment@85] - Assigned port 11224 from range 11221 - 32767.
2016-07-21 04:24:14,218 [myid:] - INFO [main:PortAssignment@85] - Assigned port 11225 from range 11221 - 32767.
2016-07-21 04:24:14,218 [myid:] - INFO [main:PortAssignment@85] - Assigned port 11226 from range 11221 - 32767.
2016-07-21 04:24:14,219 [myid:] - INFO [main:PortAssignment@85] - Assigned port 11227 from range 11221 - 32767.
2016-07-21 04:24:14,219 [myid:] - INFO [main:PortAssignment@85] - Assigned port 11228 from range 11221 - 32767.
2016-07-21 04:24:14,220 [myid:] - INFO [main:PortAssignment@85] - Assigned port 11229 from range 11221 - 32767.
2016-07-21 04:24:14,220 [myid:] - INFO [main:PortAssignment@85] - Assigned port 11230 from range 11221 - 32767.
2016-07-21 04:24:14,221 [myid:] - INFO [main:PortAssignment@85] - Assigned port 11231 from range 11221 - 32767.
2016-07-21 04:24:14,224 [myid:] - INFO [main:PortAssignment@85] - Assigned port 11232 from range 11221 - 32767.
2016-07-21 04:24:14,224 [myid:] - INFO [main:PortAssignment@85] - Assigned port 11233 from range 11221 - 32767.
2016-07-21 04:24:14,225 [myid:] - INFO [main:PortAssignment@85] - Assigned port 11234 from range 11221 - 32767.
2016-07-21 04:24:14,225 [myid:] - INFO [main:PortAssignment@85] - Assigned port 11235 from range 11221 - 32767.
2016-07-21 04:24:14,225 [myid:] - INFO [main:PortAssignment@85] - Assigned port 11236 from range 11221 - 32767.
2016-07-21 04:24:14,226 [myid:] - INFO [main:PortAssignment@85] - Assigned port 11237 from range 11221 - 32767.
2016-07-21 04:24:14,226 [myid:] - INFO [main:PortAssignment@85] - Assigned port 11238 from range 11221 - 32767.
2016-07-21 04:24:14,227 [myid:] - INFO [main:PortAssignment@85] - Assigned port 11239 from range 11221 - 32767.
2016-07-21 04:24:14,227 [myid:] - INFO [main:PortAssignment@85] - Assigned port 11240 from range 11221 - 32767.
2016-07-21 04:24:14,228 [myid:] - INFO [main:PortAssignment@85] - Assigned port 11241 from range 11221 - 32767.
2016-07-21 04:24:14,228 [myid:] - INFO [main:PortAssignment@85] - Assigned port 11242 from range 11221 - 32767.
2016-07-21 04:24:14,229 [myid:] - INFO [main:PortAssignment@85] - Assigned port 11243 from range 11221 - 32767.
2016-07-21 04:24:14,229 [myid:] - INFO [main:PortAssignment@85] - Assigned port 11244 from range 11221 - 32767.
2016-07-21 04:24:14,229 [myid:] - INFO [main:PortAssignment@85] - Assigned port 11245 from range 11221 - 32767.
2016-07-21 04:24:14,230 [myid:] - INFO [main:PortAssignment@85] - Assigned port 11246 from range 11221 - 32767.
2016-07-21 04:24:14,230 [myid:] - INFO [main:PortAssignment@85] - Assigned port 11247 from range 11221 - 32767.
2016-07-21 04:24:14,231 [myid:] - INFO [main:PortAssignment@85] - Assigned port 11248 from range 11221 - 32767.
2016-07-21 04:24:14,235 [myid:] - INFO [main:QuorumHierarchical@136] - 9, 9, 3
2016-07-21 04:24:14,262 [myid:] - INFO [main:NIOServerCnxnFactory@673] - Configuring NIO connection handler with 10s sessionless connection timeout, 1 selector thread(s), 8 worker threads, and 64 kB direct buffers.
2016-07-21 04:24:14,273 [myid:] - INFO [main:NIOServerCnxnFactory@686] - binding to port /127.0.0.1:11224
2016-07-21 04:24:14,309 [myid:] - INFO [main:QuorumPeer@776] - currentEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
2016-07-21 04:24:14,315 [myid:] - INFO [main:QuorumPeer@791] - acceptedEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
2016-07-21 04:24:14,328 [myid:] - INFO [QuorumPeerListener:QuorumCnxManager$Listener@632] - My election bind port: /127.0.0.1:11223
2016-07-21 04:24:14,329 [myid:] - INFO [main:FLEZeroWeightTest$LEThread@101] - Constructor: Thread-0
2016-07-21 04:24:14,329 [myid:] - INFO [Thread-0:FLEZeroWeightTest$LEThread@112] - Going to call leader election.
2016-07-21 04:24:14,330 [myid:] - INFO [main:QuorumHierarchical@136] - 9, 9, 3
2016-07-21 04:24:14,331 [myid:] - INFO [main:NIOServerCnxnFactory@673] - Configuring NIO connection handler with 10s sessionless connection timeout, 1 selector thread(s), 8 worker threads, and 64 kB direct buffers.
2016-07-21 04:24:14,331 [myid:] - INFO [main:NIOServerCnxnFactory@686] - binding to port /127.0.0.1:11227
2016-07-21 04:24:14,332 [myid:] - INFO [main:QuorumPeer@776] - currentEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
2016-07-21 04:24:14,351 [myid:] - INFO [main:QuorumPeer@791] - acceptedEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
2016-07-21 04:24:14,353 [myid:] - INFO [main:FLEZeroWeightTest$LEThread@101] - Constructor: Thread-2
2016-07-21 04:24:14,353 [myid:] - INFO [QuorumPeerListener:QuorumCnxManager$Listener@632] - My election bind port: /127.0.0.1:11226
2016-07-21 04:24:14,354 [myid:] - INFO [Thread-2:FLEZeroWeightTest$LEThread@112] - Going to call leader election.
2016-07-21 04:24:14,355 [myid:] - INFO [main:QuorumHierarchical@136] - 9, 9, 3
2016-07-21 04:24:14,355 [myid:] - INFO [main:NIOServerCnxnFactory@673] - Configuring NIO connection handler with 10s sessionless connection timeout, 1 selector thread(s), 8 worker threads, and 64 kB direct buffers.
2016-07-21 04:24:14,356 [myid:] - INFO [main:NIOServerCnxnFactory@686] - binding to port /127.0.0.1:11230
2016-07-21 04:24:14,357 [myid:] - INFO [main:QuorumPeer@776] - currentEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
2016-07-21 04:24:14,359 [myid:] - INFO [main:QuorumPeer@791] - acceptedEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
2016-07-21 04:24:14,368 [myid:] - INFO [main:FLEZeroWeightTest$LEThread@101] - Constructor: Thread-3
2016-07-21 04:24:14,368 [myid:] - INFO [QuorumPeerListener:QuorumCnxManager$Listener@632] - My election bind port: /127.0.0.1:11229
2016-07-21 04:24:14,369 [myid:] - INFO [Thread-3:FLEZeroWeightTest$LEThread@112] - Going to call leader election.
2016-07-21 04:24:14,370 [myid:] - INFO [main:QuorumHierarchical@136] - 9, 9, 3
2016-07-21 04:24:14,370 [myid:] - INFO [main:NIOServerCnxnFactory@673] - Configuring NIO connection handler with 10s sessionless connection timeout, 1 selector thread(s), 8 worker threads, and 64 kB direct buffers.
2016-07-21 04:24:14,371 [myid:] - INFO [main:NIOServerCnxnFactory@686] - binding to port /127.0.0.1:11233
2016-07-21 04:24:14,372 [myid:] - INFO [main:QuorumPeer@776] - currentEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
2016-07-21 04:24:14,373 [myid:] - INFO [main:QuorumPeer@791] - acceptedEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
2016-07-21 04:24:14,376 [myid:] - INFO [main:FLEZeroWeightTest$LEThread@101] - Constructor: Thread-4
2016-07-21 04:24:14,376 [myid:] - INFO [QuorumPeerListener:QuorumCnxManager$Listener@632] - My election bind port: /127.0.0.1:11232
2016-07-21 04:24:14,376 [myid:] - INFO [Thread-4:FLEZeroWeightTest$LEThread@112] - Going to call leader election.
2016-07-21 04:24:14,377 [myid:] - INFO [main:QuorumHierarchical@136] - 9, 9, 3
2016-07-21 04:24:14,378 [myid:] - INFO [main:NIOServerCnxnFactory@673] - Configuring NIO connection handler with 10s sessionless connection timeout, 1 selector thread(s), 8 worker threads, and 64 kB direct buffers.
2016-07-21 04:24:14,378 [myid:] - INFO [main:NIOServerCnxnFactory@686] - binding to port /127.0.0.1:11236
2016-07-21 04:24:14,379 [myid:] - INFO [main:QuorumPeer@776] - currentEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
2016-07-21 04:24:14,381 [myid:] - INFO [main:QuorumPeer@791] - acceptedEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
2016-07-21 04:24:14,383 [myid:] - INFO [main:FLEZeroWeightTest$LEThread@101] - Constructor: Thread-5
2016-07-21 04:24:14,383 [myid:] - INFO [QuorumPeerListener:QuorumCnxManager$Listener@632] - My election bind port: /127.0.0.1:11235
2016-07-21 04:24:14,383 [myid:] - INFO [Thread-5:FLEZeroWeightTest$LEThread@112] - Going to call leader election.
2016-07-21 04:24:14,384 [myid:] - INFO [main:QuorumHierarchical@136] - 9, 9, 3
2016-07-21 04:24:14,385 [myid:] - INFO [main:NIOServerCnxnFactory@673] - Configuring NIO connection handler with 10s sessionless connection timeout, 1 selector thread(s), 8 worker threads, and 64 kB direct buffers.
2016-07-21 04:24:14,385 [myid:] - INFO [main:NIOServerCnxnFactory@686] - binding to port /127.0.0.1:11239
2016-07-21 04:24:14,386 [myid:] - INFO [main:QuorumPeer@776] - currentEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
2016-07-21 04:24:14,388 [myid:] - INFO [main:QuorumPeer@791] - acceptedEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
2016-07-21 04:24:14,390 [myid:] - INFO [main:FLEZeroWeightTest$LEThread@101] - Constructor: Thread-6
2016-07-21 04:24:14,391 [myid:] - INFO [QuorumPeerListener:QuorumCnxManager$Listener@632] - My election bind port: /127.0.0.1:11238
2016-07-21 04:24:14,391 [myid:] - INFO [Thread-6:FLEZeroWeightTest$LEThread@112] - Going to call leader election.
2016-07-21 04:24:14,392 [myid:] - INFO [main:QuorumHierarchical@136] - 9, 9, 3
2016-07-21 04:24:14,392 [myid:] - INFO [main:NIOServerCnxnFactory@673] - Configuring NIO connection handler with 10s sessionless connection timeout, 1 selector thread(s), 8 worker threads, and 64 kB direct buffers.
2016-07-21 04:24:14,392 [myid:] - INFO [main:NIOServerCnxnFactory@686] - binding to port /127.0.0.1:11242
2016-07-21 04:24:14,393 [myid:] - INFO [main:QuorumPeer@776] - currentEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
2016-07-21 04:24:14,395 [myid:] - INFO [main:QuorumPeer@791] - acceptedEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
2016-07-21 04:24:14,397 [myid:] - INFO [main:FLEZeroWeightTest$LEThread@101] - Constructor: Thread-7
2016-07-21 04:24:14,398 [myid:] - INFO [QuorumPeerListener:QuorumCnxManager$Listener@632] - My election bind port: /127.0.0.1:11241
2016-07-21 04:24:14,398 [myid:] - INFO [Thread-7:FLEZeroWeightTest$LEThread@112] - Going to call leader election.
2016-07-21 04:24:14,399 [myid:] - INFO [main:QuorumHierarchical@136] - 9, 9, 3
2016-07-21 04:24:14,399 [myid:] - INFO [main:NIOServerCnxnFactory@673] - Configuring NIO connection handler with 10s sessionless connection timeout, 1 selector thread(s), 8 worker threads, and 64 kB direct buffers.
2016-07-21 04:24:14,400 [myid:] - INFO [main:NIOServerCnxnFactory@686] - binding to port /127.0.0.1:11245
2016-07-21 04:24:14,400 [myid:] - INFO [main:QuorumPeer@776] - currentEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
2016-07-21 04:24:14,402 [myid:] - INFO [main:QuorumPeer@791] - acceptedEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
2016-07-21 04:24:14,404 [myid:] - INFO [QuorumPeerListener:QuorumCnxManager$Listener@632] - My election bind port: /127.0.0.1:11244
2016-07-21 04:24:14,405 [myid:] - INFO [main:FLEZeroWeightTest$LEThread@101] - Constructor: Thread-8
2016-07-21 04:24:14,405 [myid:] - INFO [Thread-8:FLEZeroWeightTest$LEThread@112] - Going to call leader election.
2016-07-21 04:24:14,406 [myid:] - INFO [main:QuorumHierarchical@136] - 9, 9, 3
2016-07-21 04:24:14,407 [myid:] - INFO [main:NIOServerCnxnFactory@673] - Configuring NIO connection handler with 10s sessionless connection timeout, 1 selector thread(s), 8 worker threads, and 64 kB direct buffers.
2016-07-21 04:24:14,407 [myid:] - INFO [main:NIOServerCnxnFactory@686] - binding to port /127.0.0.1:11248
2016-07-21 04:24:14,408 [myid:] - INFO [main:QuorumPeer@776] - currentEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
2016-07-21 04:24:14,410 [myid:] - INFO [main:QuorumPeer@791] - acceptedEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
2016-07-21 04:24:14,412 [myid:] - INFO [QuorumPeerListener:QuorumCnxManager$Listener@632] - My election bind port: /127.0.0.1:11247
2016-07-21 04:24:14,412 [myid:] - INFO [main:FLEZeroWeightTest$LEThread@101] - Constructor: Thread-9
2016-07-21 04:24:14,412 [myid:] - INFO [main:FLEZeroWeightTest@162] - Started threads testZeroWeightQuorum
2016-07-21 04:24:14,412 [myid:] - INFO [Thread-9:FLEZeroWeightTest$LEThread@112] - Going to call leader election.
2016-07-21 04:24:14,431 [myid:] - INFO [Thread-0:FastLeaderElection@894] - New election. My id = 0, proposed zxid=0x0
2016-07-21 04:24:14,431 [myid:] - WARN [Thread-2:MBeanRegistry@112] - Failed to register MBean LeaderElection
2016-07-21 04:24:14,431 [myid:] - WARN [Thread-7:MBeanRegistry@112] - Failed to register MBean LeaderElection
2016-07-21 04:24:14,432 [myid:] - WARN [Thread-3:MBeanRegistry@112] - Failed to register MBean LeaderElection
2016-07-21 04:24:14,432 [myid:] - WARN [Thread-2:FastLeaderElection@876] - Failed to register with JMX
javax.management.InstanceAlreadyExistsException: org.apache.ZooKeeperService:name0=LeaderElection
at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:437)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1898)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:966)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324)
at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
at org.apache.zookeeper.jmx.MBeanRegistry.register(MBeanRegistry.java:108)
at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:873)
at org.apache.zookeeper.test.FLEZeroWeightTest$LEThread.run(FLEZeroWeightTest.java:113)
2016-07-21 04:24:14,434 [myid:] - WARN [Thread-9:MBeanRegistry@112] - Failed to register MBean LeaderElection
2016-07-21 04:24:14,436 [myid:] - WARN [Thread-9:FastLeaderElection@876] - Failed to register with JMX
javax.management.InstanceAlreadyExistsException: org.apache.ZooKeeperService:name0=LeaderElection
at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:437)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1898)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:966)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324)
at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
at org.apache.zookeeper.jmx.MBeanRegistry.register(MBeanRegistry.java:108)
at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:873)
at org.apache.zookeeper.test.FLEZeroWeightTest$LEThread.run(FLEZeroWeightTest.java:113)
2016-07-21 04:24:14,434 [myid:] - WARN [Thread-8:MBeanRegistry@112] - Failed to register MBean LeaderElection
2016-07-21 04:24:14,437 [myid:] - WARN [Thread-8:FastLeaderElection@876] - Failed to register with JMX
javax.management.InstanceAlreadyExistsException: org.apache.ZooKeeperService:name0=LeaderElection
at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:437)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1898)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:966)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324)
at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
at org.apache.zookeeper.jmx.MBeanRegistry.register(MBeanRegistry.java:108)
at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:873)
at org.apache.zookeeper.test.FLEZeroWeightTest$LEThread.run(FLEZeroWeightTest.java:113)
2016-07-21 04:24:14,433 [myid:] - WARN [Thread-6:MBeanRegistry@112] - Failed to register MBean LeaderElection
2016-07-21 04:24:14,437 [myid:] - WARN [Thread-6:FastLeaderElection@876] - Failed to register with JMX
javax.management.InstanceAlreadyExistsException: org.apache.ZooKeeperService:name0=LeaderElection
at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:437)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1898)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:966)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324)
at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
at org.apache.zookeeper.jmx.MBeanRegistry.register(MBeanRegistry.java:108)
at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:873)
at org.apache.zookeeper.test.FLEZeroWeightTest$LEThread.run(FLEZeroWeightTest.java:113)
2016-07-21 04:24:14,433 [myid:] - WARN [Thread-7:FastLeaderElection@876] - Failed to register with JMX
javax.management.InstanceAlreadyExistsException: org.apache.ZooKeeperService:name0=LeaderElection
at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:437)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1898)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:966)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324)
at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
at org.apache.zookeeper.jmx.MBeanRegistry.register(MBeanRegistry.java:108)
at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:873)
at org.apache.zookeeper.test.FLEZeroWeightTest$LEThread.run(FLEZeroWeightTest.java:113)
2016-07-21 04:24:14,433 [myid:] - WARN [Thread-3:FastLeaderElection@876] - Failed to register with JMX
javax.management.InstanceAlreadyExistsException: org.apache.ZooKeeperService:name0=LeaderElection
at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:437)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1898)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:966)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324)
at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
at org.apache.zookeeper.jmx.MBeanRegistry.register(MBeanRegistry.java:108)
at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:873)
at org.apache.zookeeper.test.FLEZeroWeightTest$LEThread.run(FLEZeroWeightTest.java:113)
2016-07-21 04:24:14,433 [myid:] - WARN [Thread-5:MBeanRegistry@112] - Failed to register MBean LeaderElection
2016-07-21 04:24:14,439 [myid:] - WARN [Thread-5:FastLeaderElection@876] - Failed to register with JMX
javax.management.InstanceAlreadyExistsException: org.apache.ZooKeeperService:name0=LeaderElection
at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:437)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1898)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:966)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324)
at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
at org.apache.zookeeper.jmx.MBeanRegistry.register(MBeanRegistry.java:108)
at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:873)
at org.apache.zookeeper.test.FLEZeroWeightTest$LEThread.run(FLEZeroWeightTest.java:113)
2016-07-21 04:24:14,432 [myid:] - WARN [Thread-4:MBeanRegistry@112] - Failed to register MBean LeaderElection
2016-07-21 04:24:14,439 [myid:] - WARN [Thread-4:FastLeaderElection@876] - Failed to register with JMX
javax.management.InstanceAlreadyExistsException: org.apache.ZooKeeperService:name0=LeaderElection
at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:437)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1898)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:966)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324)
at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
at org.apache.zookeeper.jmx.MBeanRegistry.register(MBeanRegistry.java:108)
at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:873)
at org.apache.zookeeper.test.FLEZeroWeightTest$LEThread.run(FLEZeroWeightTest.java:113)
2016-07-21 04:24:14,439 [myid:] - INFO [Thread-5:FastLeaderElection@894] - New election. My id = 4, proposed zxid=0x0
2016-07-21 04:24:14,439 [myid:] - INFO [Thread-3:FastLeaderElection@894] - New election. My id = 2, proposed zxid=0x0
2016-07-21 04:24:14,438 [myid:] - INFO [Thread-7:FastLeaderElection@894] - New election. My id = 6, proposed zxid=0x0
2016-07-21 04:24:14,438 [myid:] - INFO [WorkerReceiver[myid=0]:QuorumHierarchical@136] - 9, 9, 3
2016-07-21 04:24:14,438 [myid:] - INFO [Thread-6:FastLeaderElection@894] - New election. My id = 5, proposed zxid=0x0
2016-07-21 04:24:14,440 [myid:] - INFO [WorkerReceiver[myid=0]:FastLeaderElection@688] - Notification: 2 (message format version), 0 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 0 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version)
2016-07-21 04:24:14,437 [myid:] - INFO [Thread-8:FastLeaderElection@894] - New election. My id = 7, proposed zxid=0x0
2016-07-21 04:24:14,437 [myid:] - INFO [Thread-9:FastLeaderElection@894] - New election. My id = 8, proposed zxid=0x0
2016-07-21 04:24:14,436 [myid:] - INFO [Thread-2:FastLeaderElection@894] - New election. My id = 1, proposed zxid=0x0
2016-07-21 04:24:14,439 [myid:] - INFO [Thread-4:FastLeaderElection@894] - New election. My id = 3, proposed zxid=0x0
2016-07-21 04:24:14,446 [myid:] - INFO [/127.0.0.1:11226:QuorumCnxManager$Listener@638] - Received connection request /127.0.0.1:51059
2016-07-21 04:24:14,446 [myid:] - INFO [/127.0.0.1:11223:QuorumCnxManager$Listener@638] - Received connection request /127.0.0.1:42458
2016-07-21 04:24:14,447 [myid:] - INFO [WorkerSender[myid=0]:QuorumCnxManager@276] - Have smaller server identifier, so dropping the connection: (1, 0)
2016-07-21 04:24:14,449 [myid:] - INFO [/127.0.0.1:11229:QuorumCnxManager$Listener@638] - Received connection request /127.0.0.1:60297
2016-07-21 04:24:14,449 [myid:] - INFO [WorkerSender[myid=0]:QuorumCnxManager@276] - Have smaller server identifier, so dropping the connection: (2, 0)
2016-07-21 04:24:14,450 [myid:] - INFO [/127.0.0.1:11232:QuorumCnxManager$Listener@638] - Received connection request /127.0.0.1:45441
2016-07-21 04:24:14,450 [myid:] - WARN [SendWorker:0:QuorumCnxManager$SendWorker@832] - Interrupted while waiting for message on queue
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:982)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:63)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:820)
2016-07-21 04:24:14,452 [myid:] - INFO [/127.0.0.1:11226:QuorumCnxManager$Listener@638] - Received connection request /127.0.0.1:51068
2016-07-21 04:24:14,451 [myid:] - INFO [WorkerSender[myid=0]:QuorumCnxManager@276] - Have smaller server identifier, so dropping the connection: (3, 0)
2016-07-21 04:24:14,452 [myid:] - WARN [SendWorker:0:QuorumCnxManager$SendWorker@841] - Send worker leaving thread id 0 my id = 1
2016-07-21 04:24:14,453 [myid:] - INFO [WorkerReceiver[myid=1]:QuorumHierarchical@136] - 9, 9, 3
2016-07-21 04:24:14,454 [myid:] - INFO [WorkerSender[myid=1]:QuorumCnxManager@276] - Have smaller server identifier, so dropping the connection: (2, 1)
2016-07-21 04:24:14,455 [myid:] - INFO [WorkerSender[myid=1]:QuorumCnxManager@276] - Have smaller server identifier, so dropping the connection: (3, 1)
2016-07-21 04:24:14,456 [myid:] - INFO [/127.0.0.1:11235:QuorumCnxManager$Listener@638] - Received connection request /127.0.0.1:59529
2016-07-21 04:24:14,456 [myid:] - INFO [WorkerSender[myid=0]:QuorumCnxManager@276] - Have smaller server identifier, so dropping the connection: (4, 0)
2016-07-21 04:24:14,457 [myid:] - INFO [/127.0.0.1:11238:QuorumCnxManager$Listener@638] - Received connection request /127.0.0.1:43251
2016-07-21 04:24:14,457 [myid:] - WARN [RecvWorker:0:QuorumCnxManager$RecvWorker@915] - Connection broken for id 0, my id = 4, error =
java.net.SocketException: Socket closed
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at java.net.SocketInputStream.read(SocketInputStream.java:210)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:900)
2016-07-21 04:24:14,457 [myid:] - WARN [RecvWorker:0:QuorumCnxManager$RecvWorker@918] - Interrupting SendWorker
2016-07-21 04:24:14,457 [myid:] - INFO [WorkerSender[myid=0]:QuorumCnxManager@276] - Have smaller server identifier, so dropping the connection: (5, 0)
2016-07-21 04:24:14,458 [myid:] - INFO [/127.0.0.1:11241:QuorumCnxManager$Listener@638] - Received connection request /127.0.0.1:44371
2016-07-21 04:24:14,458 [myid:] - INFO [WorkerSender[myid=0]:QuorumCnxManager@276] - Have smaller server identifier, so dropping the connection: (6, 0)
2016-07-21 04:24:14,459 [myid:] - INFO [/127.0.0.1:11244:QuorumCnxManager$Listener@638] - Received connection request /127.0.0.1:40443
2016-07-21 04:24:14,459 [myid:] - INFO [WorkerSender[myid=0]:QuorumCnxManager@276] - Have smaller server identifier, so dropping the connection: (7, 0)
2016-07-21 04:24:14,460 [myid:] - INFO [/127.0.0.1:11247:QuorumCnxManager$Listener@638] - Received connection request /127.0.0.1:57714
2016-07-21 04:24:14,460 [myid:] - INFO [WorkerSender[myid=0]:QuorumCnxManager@276] - Have smaller server identifier, so dropping the connection: (8, 0)
2016-07-21 04:24:14,460 [myid:] - INFO [WorkerSender[myid=1]:QuorumCnxManager@276] - Have smaller server identifier, so dropping the connection: (5, 1)
2016-07-21 04:24:14,461 [myid:] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@688] - Notification: 2 (message format version), 1 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version)
2016-07-21 04:24:14,461 [myid:] - INFO [WorkerSender[myid=1]:QuorumCnxManager@276] - Have smaller server identifier, so dropping the connection: (6, 1)
2016-07-21 04:24:14,461 [myid:] - INFO [WorkerSender[myid=1]:QuorumCnxManager@276] - Have smaller server identifier, so dropping the connection: (7, 1)
2016-07-21 04:24:14,462 [myid:] - INFO [WorkerSender[myid=1]:QuorumCnxManager@276] - Have smaller server identifier, so dropping the connection: (8, 1)
2016-07-21 04:24:14,466 [myid:] - WARN [RecvWorker:0:QuorumCnxManager$RecvWorker@918] - Interrupting SendWorker
2016-07-21 04:24:14,469 [myid:] - INFO [/127.0.0.1:11226:QuorumCnxManager$Listener@638] - Received connection request /127.0.0.1:51082
2016-07-21 04:24:14,475 [myid:] - WARN [SendWorker:0:QuorumCnxManager$SendWorker@832] - Interrupted while waiting for message on queue
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:982)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:63)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:820)
2016-07-21 04:24:14,476 [myid:] - WARN [SendWorker:0:QuorumCnxManager$SendWorker@841] - Send worker leaving thread id 0 my id = 4
2016-07-21 04:24:14,479 [myid:] - INFO [WorkerReceiver[myid=1]:QuorumHierarchical@136] - 9, 9, 3
2016-07-21 04:24:14,479 [myid:] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@688] - Notification: 2 (message format version), 4 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 4 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version)
2016-07-21 04:24:14,482 [myid:] - INFO [WorkerReceiver[myid=4]:QuorumHierarchical@136] - 9, 9, 3
2016-07-21 04:24:14,482 [myid:] - INFO [WorkerReceiver[myid=4]:FastLeaderElection@688] - Notification: 2 (message format version), 1 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version)
2016-07-21 04:24:14,484 [myid:] - INFO [/127.0.0.1:11238:QuorumCnxManager$Listener@638] - Received connection request /127.0.0.1:43255
2016-07-21 04:24:14,485 [myid:] - WARN [RecvWorker:1:QuorumCnxManager$RecvWorker@915] - Connection broken for id 1, my id = 5, error =
java.net.SocketException: Socket closed
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at java.net.SocketInputStream.read(SocketInputStream.java:210)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:900)
2016-07-21 04:24:14,485 [myid:] - INFO [/127.0.0.1:11232:QuorumCnxManager$Listener@638] - Received connection request /127.0.0.1:45444
2016-07-21 04:24:14,485 [myid:] - WARN [RecvWorker:1:QuorumCnxManager$RecvWorker@918] - Interrupting SendWorker
2016-07-21 04:24:14,486 [myid:] - WARN [RecvWorker:1:QuorumCnxManager$RecvWorker@915] - Connection broken for id 1, my id = 3, error =
java.net.SocketException: Socket closed
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at java.net.SocketInputStream.read(SocketInputStream.java:210)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:900)
2016-07-21 04:24:14,486 [myid:] - WARN [RecvWorker:1:QuorumCnxManager$RecvWorker@918] - Interrupting SendWorker
2016-07-21 04:24:14,487 [myid:] - INFO [WorkerSender[myid=5]:QuorumCnxManager@276] - Have smaller server identifier, so dropping the connection: (6, 5)
2016-07-21 04:24:14,487 [myid:] - INFO [WorkerSender[myid=5]:QuorumCnxManager@276] - Have smaller server identifier, so dropping the connection: (7, 5)
2016-07-21 04:24:14,489 [myid:] - INFO [WorkerReceiver[myid=5]:QuorumHierarchical@136] - 9, 9, 3
2016-07-21 04:24:14,490 [myid:] - INFO [WorkerSender[myid=5]:QuorumCnxManager@276] - Have smaller server identifier, so dropping the connection: (8, 5)
2016-07-21 04:24:14,490 [myid:] - WARN [SendWorker:1:QuorumCnxManager$SendWorker@832] - Interrupted while waiting for message on queue
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:982)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:63)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:820)
2016-07-21 04:24:14,490 [myid:] - WARN [SendWorker:1:QuorumCnxManager$SendWorker@841] - Send worker leaving thread id 1 my id = 3
2016-07-21 04:24:14,490 [myid:] - WARN [SendWorker:1:QuorumCnxManager$SendWorker@832] - Interrupted while waiting for message on queue
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:982)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:63)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:820)
2016-07-21 04:24:14,490 [myid:] - WARN [SendWorker:1:QuorumCnxManager$SendWorker@841] - Send worker leaving thread id 1 my id = 5
2016-07-21 04:24:14,491 [myid:] - INFO [/127.0.0.1:11241:QuorumCnxManager$Listener@638] - Received connection request /127.0.0.1:44375
2016-07-21 04:24:14,491 [myid:] - INFO [WorkerReceiver[myid=5]:FastLeaderElection@688] - Notification: 2 (message format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 5 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version)
2016-07-21 04:24:14,491 [myid:] - WARN [RecvWorker:1:QuorumCnxManager$RecvWorker@915] - Connection broken for id 1, my id = 6, error =
java.net.SocketException: Socket closed
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at java.net.SocketInputStream.read(SocketInputStream.java:210)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:900)
2016-07-21 04:24:14,492 [myid:] - WARN [RecvWorker:1:QuorumCnxManager$RecvWorker@918] - Interrupting SendWorker
2016-07-21 04:24:14,492 [myid:] - INFO [/127.0.0.1:11244:QuorumCnxManager$Listener@638] - Received connection request /127.0.0.1:40447
2016-07-21 04:24:14,493 [myid:] - WARN [RecvWorker:1:QuorumCnxManager$RecvWorker@915] - Connection broken for id 1, my id = 7, error =
java.net.SocketException: Socket closed
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at java.net.SocketInputStream.read(SocketInputStream.java:210)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:900)
2016-07-21 04:24:14,493 [myid:] - WARN [RecvWorker:1:QuorumCnxManager$RecvWorker@918] - Interrupting SendWorker
2016-07-21 04:24:14,495 [myid:] - INFO [/127.0.0.1:11235:QuorumCnxManager$Listener@638] - Received connection request /127.0.0.1:59547
2016-07-21 04:24:14,497 [myid:] - INFO [/127.0.0.1:11229:QuorumCnxManager$Listener@638] - Received connection request /127.0.0.1:60301
2016-07-21 04:24:14,498 [myid:] - INFO [/127.0.0.1:11247:QuorumCnxManager$Listener@638] - Received connection request /127.0.0.1:57718
2016-07-21 04:24:14,498 [myid:] - WARN [RecvWorker:1:QuorumCnxManager$RecvWorker@915] - Connection broken for id 1, my id = 2, error =
java.net.SocketException: Socket closed
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at java.net.SocketInputStream.read(SocketInputStream.java:210)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:900)
2016-07-21 04:24:14,499 [myid:] - WARN [RecvWorker:1:QuorumCnxManager$RecvWorker@918] - Interrupting SendWorker
2016-07-21 04:24:14,500 [myid:] - INFO [/127.0.0.1:11247:QuorumCnxManager$Listener@638] - Received connection request /127.0.0.1:57732
2016-07-21 04:24:14,500 [myid:] - WARN [SendWorker:1:QuorumCnxManager$SendWorker@832] - Interrupted while waiting for message on queue
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:982)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:63)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:820)
2016-07-21 04:24:14,500 [myid:] - INFO [WorkerSender[myid=4]:QuorumCnxManager@276] - Have smaller server identifier, so dropping the connection: (5, 4)
2016-07-21 04:24:14,501 [myid:] - WARN [SendWorker:1:QuorumCnxManager$SendWorker@841] - Send worker leaving thread id 1 my id = 7
2016-07-21 04:24:14,501 [myid:] - INFO [WorkerSender[myid=4]:QuorumCnxManager@276] - Have smaller server identifier, so dropping the connection: (6, 4)
2016-07-21 04:24:14,502 [myid:] - INFO [WorkerSender[myid=4]:QuorumCnxManager@276] - Have smaller server identifier, so dropping the connection: (7, 4)
2016-07-21 04:24:14,502 [myid:] - INFO [WorkerSender[myid=4]:QuorumCnxManager@276] - Have smaller server identifier, so dropping the connection: (8, 4)
2016-07-21 04:24:14,504 [myid:] - INFO [WorkerReceiver[myid=4]:QuorumHierarchical@136] - 9, 9, 3
2016-07-21 04:24:14,504 [myid:] - INFO [WorkerReceiver[myid=4]:FastLeaderElection@688] - Notification: 2 (message format version), 4 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 4 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version)
2016-07-21 04:24:14,505 [myid:] - WARN [SendWorker:1:QuorumCnxManager$SendWorker@832] - Interrupted while waiting for message on queue
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:982)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:63)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:820)
2016-07-21 04:24:14,505 [myid:] - WARN [SendWorker:1:QuorumCnxManager$SendWorker@841] - Send worker leaving thread id 1 my id = 6
2016-07-21 04:24:14,505 [myid:] - INFO [/127.0.0.1:11238:QuorumCnxManager$Listener@638] - Received connection request /127.0.0.1:43280
2016-07-21 04:24:14,506 [myid:] - INFO [/127.0.0.1:11223:QuorumCnxManager$Listener@638] - Received connection request /127.0.0.1:42457
2016-07-21 04:24:14,506 [myid:] - WARN [RecvWorker:4:QuorumCnxManager$RecvWorker@915] - Connection broken for id 4, my id = 5, error =
java.net.SocketException: Socket closed
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at java.net.SocketInputStream.read(SocketInputStream.java:210)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:900)
2016-07-21 04:24:14,506 [myid:] - WARN [RecvWorker:4:QuorumCnxManager$RecvWorker@918] - Interrupting SendWorker
2016-07-21 04:24:14,507 [myid:] - INFO [/127.0.0.1:11226:QuorumCnxManager$Listener@638] - Received connection request /127.0.0.1:51083
2016-07-21 04:24:14,507 [myid:] - WARN [SendWorker:3:QuorumCnxManager$SendWorker@837] - Exception when using channel: for id 3 my id = 1 error = java.net.SocketException: Broken pipe
2016-07-21 04:24:14,507 [myid:] - WARN [RecvWorker:3:QuorumCnxManager$RecvWorker@918] - Interrupting SendWorker
2016-07-21 04:24:14,508 [myid:] - INFO [/127.0.0.1:11226:QuorumCnxManager$Listener@638] - Received connection request /127.0.0.1:51084
2016-07-21 04:24:14,508 [myid:] - INFO [WorkerReceiver[myid=0]:QuorumHierarchical@136] - 9, 9, 3
2016-07-21 04:24:14,508 [myid:] - WARN [SendWorker:6:QuorumCnxManager$SendWorker@837] - Exception when using channel: for id 6 my id = 1 error = java.net.SocketException: Broken pipe
2016-07-21 04:24:14,508 [myid:] - INFO [WorkerReceiver[myid=0]:FastLeaderElection@688] - Notification: 2 (message format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 5 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version)
2016-07-21 04:24:14,508 [myid:] - INFO [WorkerReceiver[myid=5]:QuorumHierarchical@136] - 9, 9, 3
2016-07-21 04:24:14,509 [myid:] - INFO [WorkerReceiver[myid=5]:FastLeaderElection@688] - Notification: 2 (message format version), 0 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 0 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version)
2016-07-21 04:24:14,510 [myid:] - INFO [WorkerReceiver[myid=1]:QuorumHierarchical@136] - 9, 9, 3
2016-07-21 04:24:14,510 [myid:] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@688] - Notification: 2 (message format version), 6 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 6 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version)
2016-07-21 04:24:14,510 [myid:] - WARN [RecvWorker:6:QuorumCnxManager$RecvWorker@918] - Interrupting SendWorker
2016-07-21 04:24:14,516 [myid:] - INFO [WorkerSender[myid=2]:QuorumCnxManager@276] - Have smaller server identifier, so dropping the connection: (3, 2)
2016-07-21 04:24:14,516 [myid:] - WARN [SendWorker:1:QuorumCnxManager$SendWorker@832] - Interrupted while waiting for message on queue
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:982)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:63)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:820)
2016-07-21 04:24:14,517 [myid:] - WARN [SendWorker:1:QuorumCnxManager$SendWorker@841] - Send worker leaving thread id 1 my id = 2
2016-07-21 04:24:14,517 [myid:] - INFO [WorkerReceiver[myid=2]:QuorumHierarchical@136] - 9, 9, 3
2016-07-21 04:24:14,517 [myid:] - WARN [SendWorker:5:QuorumCnxManager$SendWorker@837] - Exception when using channel: for id 5 my id = 4 error = java.net.SocketException: Broken pipe
2016-07-21 04:24:14,518 [myid:] - INFO [/127.0.0.1:11235:QuorumCnxManager$Listener@638] - Received connection request /127.0.0.1:59564
2016-07-21 04:24:14,518 [myid:] - INFO [WorkerSender[myid=2]:QuorumCnxManager@276] - Have smaller server identifier, so dropping the connection: (4, 2)
2016-07-21 04:24:14,518 [myid:] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@688] - Notification: 2 (message format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version)
2016-07-21 04:24:14,518 [myid:] - WARN [RecvWorker:5:QuorumCnxManager$RecvWorker@918] - Interrupting SendWorker
2016-07-21 04:24:14,520 [myid:] - INFO [/127.0.0.1:11232:QuorumCnxManager$Listener@638] - Received connection request /127.0.0.1:45462
2016-07-21 04:24:14,520 [myid:] - WARN [SendWorker:6:QuorumCnxManager$SendWorker@841] - Send worker leaving thread id 6 my id = 1
2016-07-21 04:24:14,520 [myid:] - INFO [/127.0.0.1:11232:QuorumCnxManager$Listener@638] - Received connection request /127.0.0.1:45470
2016-07-21 04:24:14,521 [myid:] - WARN [SendWorker:3:QuorumCnxManager$SendWorker@841] - Send worker leaving thread id 3 my id = 1
2016-07-21 04:24:14,521 [myid:] - INFO [WorkerReceiver[myid=3]:QuorumHierarchical@136] - 9, 9, 3
2016-07-21 04:24:14,521 [myid:] - INFO [/127.0.0.1:11226:QuorumCnxManager$Listener@638] - Received connection request /127.0.0.1:51085
2016-07-21 04:24:14,522 [myid:] - INFO [WorkerSender[myid=3]:QuorumCnxManager@276] - Have smaller server identifier, so dropping the connection: (4, 3)
2016-07-21 04:24:14,522 [myid:] - INFO [WorkerReceiver[myid=3]:FastLeaderElection@688] - Notification: 2 (message format version), 3 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 3 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version)
2016-07-21 04:24:14,522 [myid:] - WARN [RecvWorker:5:QuorumCnxManager$RecvWorker@915] - Connection broken for id 5, my id = 1, error =
java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:392)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:900)
2016-07-21 04:24:14,522 [myid:] - WARN [SendWorker:5:QuorumCnxManager$SendWorker@837] - Exception when using channel: for id 5 my id = 1 error = java.net.SocketException: Broken pipe
2016-07-21 04:24:14,523 [myid:] - WARN [SendWorker:5:QuorumCnxManager$SendWorker@841] - Send worker leaving thread id 5 my id = 1
2016-07-21 04:24:14,522 [myid:] - WARN [SendWorker:4:QuorumCnxManager$SendWorker@832] - Interrupted while waiting for message on queue
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:982)
at org.apache.zoo
...[truncated 231211 chars]...
.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:982)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:63)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:820)
2016-07-21 04:24:40,861 [myid:] - WARN [SendWorker:3:QuorumCnxManager$SendWorker@841] - Send worker leaving thread id 3 my id = 6
2016-07-21 04:24:40,805 [myid:] - WARN [SendWorker:4:QuorumCnxManager$SendWorker@841] - Send worker leaving thread id 4 my id = 3
2016-07-21 04:24:40,805 [myid:] - INFO [WorkerReceiver[myid=8]:QuorumHierarchical@136] - 9, 9, 3
2016-07-21 04:24:40,805 [myid:] - WARN [SendWorker:3:QuorumCnxManager$SendWorker@841] - Send worker leaving thread id 3 my id = 7
2016-07-21 04:24:40,804 [myid:] - INFO [WorkerReceiver[myid=6]:FastLeaderElection@688] - Notification: 2 (message format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 5 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version)
2016-07-21 04:24:40,804 [myid:] - INFO [WorkerReceiver[myid=4]:FastLeaderElection@688] - Notification: 2 (message format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 5 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version)
2016-07-21 04:24:40,862 [myid:] - INFO [WorkerReceiver[myid=8]:FastLeaderElection@688] - Notification: 2 (message format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 5 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version)
2016-07-21 04:24:40,861 [myid:] - INFO [WorkerReceiver[myid=5]:FastLeaderElection@688] - Notification: 2 (message format version), 2 (n.leader), 0x0 (n.zxid), 0xffffffffffffffff (n.round), LEADING (n.state), 2 (n.sid), 0x0 (n.peerEPoch), FOLLOWING (my state)0 (n.config version)
2016-07-21 04:24:40,860 [myid:] - INFO [/127.0.0.1:11235:QuorumCnxManager$Listener@661] - Leaving listener
2016-07-21 04:24:40,859 [myid:] - INFO [WorkerReceiver[myid=7]:FastLeaderElection@688] - Notification: 2 (message format version), 4 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 4 (n.sid), 0x0 (n.peerEPoch), FOLLOWING (my state)0 (n.config version)
2016-07-21 04:24:40,863 [myid:] - INFO [main:QuorumBase@398] - Shutting down leader election QuorumPeer
2016-07-21 04:24:40,863 [myid:] - INFO [main:QuorumBase@403] - Waiting for QuorumPeer to exit thread
2016-07-21 04:24:40,863 [myid:] - WARN [SendWorker:7:QuorumCnxManager$SendWorker@832] - Interrupted while waiting for message on queue
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:982)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:63)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:820)
2016-07-21 04:24:40,864 [myid:] - WARN [RecvWorker:4:QuorumCnxManager$RecvWorker@915] - Connection broken for id 4, my id = 7, error =
java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:392)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:900)
2016-07-21 04:24:40,864 [myid:] - WARN [RecvWorker:5:QuorumCnxManager$RecvWorker@915] - Connection broken for id 5, my id = 4, error =
java.net.SocketException: Socket closed
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at java.net.SocketInputStream.read(SocketInputStream.java:210)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:900)
2016-07-21 04:24:40,864 [myid:] - WARN [RecvWorker:5:QuorumCnxManager$RecvWorker@918] - Interrupting SendWorker
2016-07-21 04:24:40,864 [myid:] - WARN [SendWorker:4:QuorumCnxManager$SendWorker@837] - Exception when using channel: for id 4 my id = 7 error = java.net.SocketException: Broken pipe
2016-07-21 04:24:40,865 [myid:] - INFO [WorkerReceiver[myid=8]:QuorumHierarchical@136] - 9, 9, 3
2016-07-21 04:24:40,862 [myid:] - INFO [WorkerReceiver[myid=4]:QuorumHierarchical@136] - 9, 9, 3
2016-07-21 04:24:40,862 [myid:] - INFO [WorkerReceiver[myid=6]:QuorumHierarchical@136] - 9, 9, 3
2016-07-21 04:24:40,865 [myid:] - INFO [WorkerReceiver[myid=8]:FastLeaderElection@688] - Notification: 2 (message format version), 2 (n.leader), 0x0 (n.zxid), 0xffffffffffffffff (n.round), LEADING (n.state), 2 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version)
2016-07-21 04:24:40,865 [myid:] - INFO [main:QuorumBase@394] - Shutting down quorum peer QuorumPeer
2016-07-21 04:24:40,865 [myid:] - INFO [WorkerReceiver[myid=4]:FastLeaderElection@688] - Notification: 2 (message format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 5 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version)
2016-07-21 04:24:40,865 [myid:] - WARN [SendWorker:4:QuorumCnxManager$SendWorker@841] - Send worker leaving thread id 4 my id = 7
2016-07-21 04:24:40,864 [myid:] - WARN [RecvWorker:7:QuorumCnxManager$RecvWorker@915] - Connection broken for id 7, my id = 4, error =
java.net.SocketException: Socket closed
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at java.net.SocketInputStream.read(SocketInputStream.java:210)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:900)
2016-07-21 04:24:40,866 [myid:] - WARN [RecvWorker:7:QuorumCnxManager$RecvWorker@918] - Interrupting SendWorker
2016-07-21 04:24:40,864 [myid:] - WARN [RecvWorker:4:QuorumCnxManager$RecvWorker@918] - Interrupting SendWorker
2016-07-21 04:24:40,864 [myid:] - INFO [WorkerReceiver[myid=7]:QuorumHierarchical@136] - 9, 9, 3
2016-07-21 04:24:40,864 [myid:] - WARN [RecvWorker:4:QuorumCnxManager$RecvWorker@915] - Connection broken for id 4, my id = 5, error =
java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:392)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:900)
2016-07-21 04:24:40,867 [myid:] - WARN [RecvWorker:4:QuorumCnxManager$RecvWorker@918] - Interrupting SendWorker
2016-07-21 04:24:40,864 [myid:] - WARN [SendWorker:5:QuorumCnxManager$SendWorker@832] - Interrupted while waiting for message on queue
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:982)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:63)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:820)
2016-07-21 04:24:40,864 [myid:] - WARN [SendWorker:7:QuorumCnxManager$SendWorker@841] - Send worker leaving thread id 7 my id = 4
2016-07-21 04:24:40,864 [myid:] - WARN [RecvWorker:8:QuorumCnxManager$RecvWorker@915] - Connection broken for id 8, my id = 4, error =
java.net.SocketException: Socket closed
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at java.net.SocketInputStream.read(SocketInputStream.java:210)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:900)
2016-07-21 04:24:40,868 [myid:] - WARN [RecvWorker:8:QuorumCnxManager$RecvWorker@918] - Interrupting SendWorker
2016-07-21 04:24:40,863 [myid:] - WARN [RecvWorker:4:QuorumCnxManager$RecvWorker@915] - Connection broken for id 4, my id = 6, error =
java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:392)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:900)
2016-07-21 04:24:40,869 [myid:] - WARN [RecvWorker:4:QuorumCnxManager$RecvWorker@918] - Interrupting SendWorker
2016-07-21 04:24:40,863 [myid:] - WARN [RecvWorker:4:QuorumCnxManager$RecvWorker@915] - Connection broken for id 4, my id = 8, error =
java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:392)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:900)
2016-07-21 04:24:40,870 [myid:] - WARN [RecvWorker:4:QuorumCnxManager$RecvWorker@918] - Interrupting SendWorker
2016-07-21 04:24:40,863 [myid:] - WARN [SendWorker:8:QuorumCnxManager$SendWorker@832] - Interrupted while waiting for message on queue
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:982)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:63)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:820)
2016-07-21 04:24:40,863 [myid:] - INFO [WorkerReceiver[myid=5]:QuorumHierarchical@136] - 9, 9, 3
2016-07-21 04:24:40,863 [myid:] - WARN [SendWorker:6:QuorumCnxManager$SendWorker@832] - Interrupted while waiting for message on queue
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:982)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:63)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:820)
2016-07-21 04:24:40,863 [myid:] - WARN [RecvWorker:6:QuorumCnxManager$RecvWorker@915] - Connection broken for id 6, my id = 4, error =
java.net.SocketException: Socket closed
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at java.net.SocketInputStream.read(SocketInputStream.java:210)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:900)
2016-07-21 04:24:40,871 [myid:] - WARN [RecvWorker:6:QuorumCnxManager$RecvWorker@918] - Interrupting SendWorker
2016-07-21 04:24:40,871 [myid:] - WARN [SendWorker:6:QuorumCnxManager$SendWorker@841] - Send worker leaving thread id 6 my id = 4
2016-07-21 04:24:40,871 [myid:] - INFO [WorkerReceiver[myid=5]:FastLeaderElection@688] - Notification: 2 (message format version), 4 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 4 (n.sid), 0x0 (n.peerEPoch), FOLLOWING (my state)0 (n.config version)
2016-07-21 04:24:40,870 [myid:] - WARN [SendWorker:8:QuorumCnxManager$SendWorker@841] - Send worker leaving thread id 8 my id = 4
2016-07-21 04:24:40,870 [myid:] - WARN [SendWorker:4:QuorumCnxManager$SendWorker@832] - Interrupted while waiting for message on queue
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:982)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:63)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:820)
2016-07-21 04:24:40,869 [myid:] - WARN [SendWorker:4:QuorumCnxManager$SendWorker@832] - Interrupted while waiting for message on queue
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:982)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:63)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:820)
2016-07-21 04:24:40,868 [myid:] - WARN [SendWorker:5:QuorumCnxManager$SendWorker@841] - Send worker leaving thread id 5 my id = 4
2016-07-21 04:24:40,868 [myid:] - WARN [SendWorker:4:QuorumCnxManager$SendWorker@832] - Interrupted while waiting for message on queue
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:982)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:63)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:820)
2016-07-21 04:24:40,867 [myid:] - INFO [WorkerReceiver[myid=7]:FastLeaderElection@688] - Notification: 2 (message format version), 8 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 8 (n.sid), 0x0 (n.peerEPoch), FOLLOWING (my state)0 (n.config version)
2016-07-21 04:24:40,866 [myid:] - INFO [/127.0.0.1:11238:QuorumCnxManager$Listener@661] - Leaving listener
2016-07-21 04:24:40,866 [myid:] - INFO [WorkerReceiver[myid=8]:QuorumHierarchical@136] - 9, 9, 3
2016-07-21 04:24:40,866 [myid:] - INFO [WorkerReceiver[myid=4]:FastLeaderElection$Messenger$WorkerReceiver@440] - WorkerReceiver is down
2016-07-21 04:24:40,874 [myid:] - INFO [WorkerReceiver[myid=7]:QuorumHierarchical@136] - 9, 9, 3
2016-07-21 04:24:40,874 [myid:] - INFO [WorkerReceiver[myid=7]:FastLeaderElection@688] - Notification: 2 (message format version), 8 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 8 (n.sid), 0x0 (n.peerEPoch), FOLLOWING (my state)0 (n.config version)
2016-07-21 04:24:40,866 [myid:] - INFO [WorkerReceiver[myid=6]:FastLeaderElection@688] - Notification: 2 (message format version), 6 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 6 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version)
2016-07-21 04:24:40,866 [myid:] - INFO [Thread-5:FLEZeroWeightTest$LEThread@115] - Thread 4 got a null vote
2016-07-21 04:24:40,875 [myid:] - INFO [WorkerReceiver[myid=7]:QuorumHierarchical@136] - 9, 9, 3
2016-07-21 04:24:40,875 [myid:] - INFO [WorkerReceiver[myid=7]:FastLeaderElection@688] - Notification: 2 (message format version), 2 (n.leader), 0x0 (n.zxid), 0xffffffffffffffff (n.round), FOLLOWING (n.state), 1 (n.sid), 0x0 (n.peerEPoch), FOLLOWING (my state)0 (n.config version)
2016-07-21 04:24:40,873 [myid:] - WARN [SendWorker:4:QuorumCnxManager$SendWorker@841] - Send worker leaving thread id 4 my id = 5
2016-07-21 04:24:40,875 [myid:] - INFO [WorkerReceiver[myid=7]:QuorumHierarchical@136] - 9, 9, 3
2016-07-21 04:24:40,875 [myid:] - INFO [WorkerReceiver[myid=7]:FastLeaderElection@688] - Notification: 2 (message format version), 2 (n.leader), 0x0 (n.zxid), 0xffffffffffffffff (n.round), FOLLOWING (n.state), 3 (n.sid), 0x0 (n.peerEPoch), FOLLOWING (my state)0 (n.config version)
2016-07-21 04:24:40,872 [myid:] - WARN [SendWorker:4:QuorumCnxManager$SendWorker@841] - Send worker leaving thread id 4 my id = 6
2016-07-21 04:24:40,872 [myid:] - INFO [WorkerReceiver[myid=5]:QuorumHierarchical@136] - 9, 9, 3
2016-07-21 04:24:40,876 [myid:] - INFO [WorkerReceiver[myid=7]:QuorumHierarchical@136] - 9, 9, 3
2016-07-21 04:24:40,876 [myid:] - INFO [WorkerReceiver[myid=7]:FastLeaderElection@688] - Notification: 2 (message format version), 8 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 8 (n.sid), 0x0 (n.peerEPoch), FOLLOWING (my state)0 (n.config version)
2016-07-21 04:24:40,876 [myid:] - WARN [WorkerSender[myid=5]:QuorumCnxManager@455] - Cannot open channel to 4 at election address /127.0.0.1:11235
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:441)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:482)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:419)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:486)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:465)
at java.lang.Thread.run(Thread.java:745)
2016-07-21 04:24:40,877 [myid:] - WARN [RecvWorker:7:QuorumCnxManager$RecvWorker@915] - Connection broken for id 7, my id = 5, error =
java.net.SocketException: Socket closed
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at java.net.SocketInputStream.read(SocketInputStream.java:210)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:900)
2016-07-21 04:24:40,877 [myid:] - WARN [RecvWorker:7:QuorumCnxManager$RecvWorker@918] - Interrupting SendWorker
2016-07-21 04:24:40,872 [myid:] - WARN [SendWorker:4:QuorumCnxManager$SendWorker@841] - Send worker leaving thread id 4 my id = 8
2016-07-21 04:24:40,877 [myid:] - INFO [WorkerReceiver[myid=5]:FastLeaderElection@688] - Notification: 2 (message format version), 2 (n.leader), 0x0 (n.zxid), 0xffffffffffffffff (n.round), FOLLOWING (n.state), 3 (n.sid), 0x0 (n.peerEPoch), FOLLOWING (my state)0 (n.config version)
2016-07-21 04:24:40,877 [myid:] - WARN [RecvWorker:5:QuorumCnxManager$RecvWorker@915] - Connection broken for id 5, my id = 8, error =
java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:392)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:900)
2016-07-21 04:24:40,877 [myid:] - WARN [RecvWorker:5:QuorumCnxManager$RecvWorker@915] - Connection broken for id 5, my id = 6, error =
java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:392)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:900)
2016-07-21 04:24:40,877 [myid:] - WARN [SendWorker:8:QuorumCnxManager$SendWorker@832] - Interrupted while waiting for message on queue
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:982)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:63)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:820)
2016-07-21 04:24:40,877 [myid:] - WARN [RecvWorker:8:QuorumCnxManager$RecvWorker@915] - Connection broken for id 8, my id = 5, error =
java.net.SocketException: Socket closed
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at java.net.SocketInputStream.read(SocketInputStream.java:210)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:900)
2016-07-21 04:24:40,879 [myid:] - WARN [RecvWorker:8:QuorumCnxManager$RecvWorker@918] - Interrupting SendWorker
2016-07-21 04:24:40,877 [myid:] - WARN [RecvWorker:5:QuorumCnxManager$RecvWorker@915] - Connection broken for id 5, my id = 7, error =
java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:392)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:900)
2016-07-21 04:24:40,877 [myid:] - WARN [SendWorker:7:QuorumCnxManager$SendWorker@832] - Interrupted while waiting for message on queue
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:982)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:63)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:820)
2016-07-21 04:24:40,877 [myid:] - WARN [SendWorker:6:QuorumCnxManager$SendWorker@832] - Interrupted while waiting for message on queue
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:982)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:63)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:820)
2016-07-21 04:24:40,877 [myid:] - WARN [RecvWorker:6:QuorumCnxManager$RecvWorker@915] - Connection broken for id 6, my id = 5, error =
java.net.SocketException: Socket closed
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at java.net.SocketInputStream.read(SocketInputStream.java:210)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:900)
2016-07-21 04:24:40,881 [myid:] - WARN [RecvWorker:6:QuorumCnxManager$RecvWorker@918] - Interrupting SendWorker
2016-07-21 04:24:40,877 [myid:] - INFO [WorkerSender[myid=5]:FastLeaderElection$Messenger$WorkerSender@470] - WorkerSender is down
2016-07-21 04:24:40,877 [myid:] - INFO [main:QuorumBase@398] - Shutting down leader election QuorumPeer
2016-07-21 04:24:40,876 [myid:] - INFO [WorkerReceiver[myid=7]:QuorumHierarchical@136] - 9, 9, 3
2016-07-21 04:24:40,876 [myid:] - INFO [Thread-9:FLEZeroWeightTest$LEThread@125] - Finished election: 8, 2
2016-07-21 04:24:40,876 [myid:] - INFO [WorkerReceiver[myid=8]:FastLeaderElection@688] - Notification: 2 (message format version), 6 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 6 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version)
2016-07-21 04:24:40,875 [myid:] - INFO [WorkerReceiver[myid=6]:QuorumHierarchical@136] - 9, 9, 3
2016-07-21 04:24:40,881 [myid:] - INFO [WorkerReceiver[myid=7]:FastLeaderElection@688] - Notification: 2 (message format version), 6 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 6 (n.sid), 0x0 (n.peerEPoch), FOLLOWING (my state)0 (n.config version)
2016-07-21 04:24:40,881 [myid:] - INFO [main:QuorumBase@403] - Waiting for QuorumPeer to exit thread
2016-07-21 04:24:40,880 [myid:] - WARN [SendWorker:6:QuorumCnxManager$SendWorker@841] - Send worker leaving thread id 6 my id = 5
2016-07-21 04:24:40,880 [myid:] - WARN [SendWorker:7:QuorumCnxManager$SendWorker@841] - Send worker leaving thread id 7 my id = 5
2016-07-21 04:24:40,880 [myid:] - WARN [RecvWorker:5:QuorumCnxManager$RecvWorker@918] - Interrupting SendWorker
2016-07-21 04:24:40,879 [myid:] - WARN [SendWorker:8:QuorumCnxManager$SendWorker@841] - Send worker leaving thread id 8 my id = 5
2016-07-21 04:24:40,878 [myid:] - WARN [RecvWorker:5:QuorumCnxManager$RecvWorker@918] - Interrupting SendWorker
2016-07-21 04:24:40,883 [myid:] - WARN [SendWorker:5:QuorumCnxManager$SendWorker@832] - Interrupted while waiting for message on queue
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:982)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:63)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:820)
2016-07-21 04:24:40,878 [myid:] - WARN [RecvWorker:5:QuorumCnxManager$RecvWorker@918] - Interrupting SendWorker
2016-07-21 04:24:40,878 [myid:] - INFO [WorkerReceiver[myid=5]:FastLeaderElection$Messenger$WorkerReceiver@440] - WorkerReceiver is down
2016-07-21 04:24:40,883 [myid:] - WARN [SendWorker:5:QuorumCnxManager$SendWorker@832] - Interrupted while waiting for message on queue
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:982)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:63)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:820)
2016-07-21 04:24:40,883 [myid:] - INFO [WorkerReceiver[myid=7]:QuorumHierarchical@136] - 9, 9, 3
2016-07-21 04:24:40,883 [myid:] - WARN [SendWorker:5:QuorumCnxManager$SendWorker@841] - Send worker leaving thread id 5 my id = 7
2016-07-21 04:24:40,883 [myid:] - WARN [SendWorker:5:QuorumCnxManager$SendWorker@832] - Interrupted while waiting for message on queue
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:982)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:63)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:820)
2016-07-21 04:24:40,882 [myid:] - INFO [main:QuorumBase@394] - Shutting down quorum peer QuorumPeer
2016-07-21 04:24:40,882 [myid:] - INFO [WorkerReceiver[myid=6]:FastLeaderElection@688] - Notification: 2 (message format version), 7 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 7 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version)
2016-07-21 04:24:40,884 [myid:] - WARN [SendWorker:5:QuorumCnxManager$SendWorker@841] - Send worker leaving thread id 5 my id = 6
2016-07-21 04:24:40,884 [myid:] - INFO [WorkerReceiver[myid=7]:FastLeaderElection@688] - Notification: 2 (message format version), 8 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 8 (n.sid), 0x0 (n.peerEPoch), FOLLOWING (my state)0 (n.config version)
2016-07-21 04:24:40,884 [myid:] - WARN [SendWorker:5:QuorumCnxManager$SendWorker@841] - Send worker leaving thread id 5 my id = 8
2016-07-21 04:24:40,883 [myid:] - INFO [WorkerReceiver[myid=8]:QuorumHierarchical@136] - 9, 9, 3
2016-07-21 04:24:40,889 [myid:] - INFO [WorkerReceiver[myid=6]:QuorumHierarchical@136] - 9, 9, 3
2016-07-21 04:24:40,885 [myid:] - INFO [/127.0.0.1:11241:QuorumCnxManager$Listener@661] - Leaving listener
2016-07-21 04:24:40,890 [myid:] - INFO [Thread-7:FLEZeroWeightTest$LEThread@115] - Thread 6 got a null vote
2016-07-21 04:24:40,891 [myid:] - WARN [SendWorker:7:QuorumCnxManager$SendWorker@832] - Interrupted while waiting for message on queue
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:982)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:63)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:820)
2016-07-21 04:24:40,891 [myid:] - WARN [SendWorker:7:QuorumCnxManager$SendWorker@841] - Send worker leaving thread id 7 my id = 6
2016-07-21 04:24:40,890 [myid:] - INFO [WorkerReceiver[myid=6]:FastLeaderElection@688] - Notification: 2 (message format version), 7 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 7 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version)
2016-07-21 04:24:40,889 [myid:] - INFO [WorkerReceiver[myid=8]:FastLeaderElection@688] - Notification: 2 (message format version), 2 (n.leader), 0x0 (n.zxid), 0xffffffffffffffff (n.round), FOLLOWING (n.state), 3 (n.sid), 0x0 (n.peerEPoch), FOLLOWING (my state)0 (n.config version)
2016-07-21 04:24:40,893 [myid:] - INFO [WorkerReceiver[myid=6]:FastLeaderElection$Messenger$WorkerReceiver@440] - WorkerReceiver is down
2016-07-21 04:24:40,892 [myid:] - WARN [SendWorker:8:QuorumCnxManager$SendWorker@832] - Interrupted while waiting for message on queue
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:982)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:63)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:820)
2016-07-21 04:24:40,892 [myid:] - WARN [RecvWorker:6:QuorumCnxManager$RecvWorker@915] - Connection broken for id 6, my id = 7, error =
java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:392)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:900)
2016-07-21 04:24:40,894 [myid:] - WARN [RecvWorker:6:QuorumCnxManager$RecvWorker@918] - Interrupting SendWorker
2016-07-21 04:24:40,891 [myid:] - WARN [RecvWorker:6:QuorumCnxManager$RecvWorker@915] - Connection broken for id 6, my id = 8, error =
java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:392)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:900)
2016-07-21 04:24:40,894 [myid:] - WARN [RecvWorker:6:QuorumCnxManager$RecvWorker@918] - Interrupting SendWorker
2016-07-21 04:24:40,891 [myid:] - WARN [RecvWorker:8:QuorumCnxManager$RecvWorker@915] - Connection broken for id 8, my id = 6, error =
java.net.SocketException: Socket closed
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at java.net.SocketInputStream.read(SocketInputStream.java:210)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:900)
2016-07-21 04:24:40,894 [myid:] - WARN [RecvWorker:8:QuorumCnxManager$RecvWorker@918] - Interrupting SendWorker
2016-07-21 04:24:40,891 [myid:] - WARN [RecvWorker:7:QuorumCnxManager$RecvWorker@915] - Connection broken for id 7, my id = 6, error =
java.net.SocketException: Socket closed
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at java.net.SocketInputStream.read(SocketInputStream.java:210)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:900)
2016-07-21 04:24:40,895 [myid:] - WARN [RecvWorker:7:QuorumCnxManager$RecvWorker@918] - Interrupting SendWorker
2016-07-21 04:24:40,891 [myid:] - INFO [main:QuorumBase@398] - Shutting down leader election QuorumPeer
2016-07-21 04:24:40,891 [myid:] - INFO [WorkerReceiver[myid=7]:QuorumHierarchical@136] - 9, 9, 3
2016-07-21 04:24:40,895 [myid:] - INFO [main:QuorumBase@403] - Waiting for QuorumPeer to exit thread
2016-07-21 04:24:40,894 [myid:] - INFO [WorkerReceiver[myid=8]:QuorumHierarchical@136] - 9, 9, 3
2016-07-21 04:24:40,894 [myid:] - WARN [SendWorker:6:QuorumCnxManager$SendWorker@832] - Interrupted while waiting for message on queue
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:982)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:63)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:820)
2016-07-21 04:24:40,894 [myid:] - WARN [SendWorker:6:QuorumCnxManager$SendWorker@832] - Interrupted while waiting for message on queue
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:982)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:63)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:820)
2016-07-21 04:24:40,896 [myid:] - WARN [SendWorker:6:QuorumCnxManager$SendWorker@841] - Send worker leaving thread id 6 my id = 7
2016-07-21 04:24:40,893 [myid:] - WARN [SendWorker:8:QuorumCnxManager$SendWorker@841] - Send worker leaving thread id 8 my id = 6
2016-07-21 04:24:40,896 [myid:] - WARN [SendWorker:6:QuorumCnxManager$SendWorker@841] - Send worker leaving thread id 6 my id = 8
2016-07-21 04:24:40,896 [myid:] - INFO [WorkerReceiver[myid=8]:FastLeaderElection@688] - Notification: 2 (message format version), 2 (n.leader), 0x0 (n.zxid), 0xffffffffffffffff (n.round), FOLLOWING (n.state), 3 (n.sid), 0x0 (n.peerEPoch), FOLLOWING (my state)0 (n.config version)
2016-07-21 04:24:40,896 [myid:] - INFO [main:QuorumBase@394] - Shutting down quorum peer QuorumPeer
2016-07-21 04:24:40,896 [myid:] - INFO [WorkerReceiver[myid=7]:FastLeaderElection@688] - Notification: 2 (message format version), 8 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 8 (n.sid), 0x0 (n.peerEPoch), FOLLOWING (my state)0 (n.config version)
2016-07-21 04:24:40,897 [myid:] - INFO [/127.0.0.1:11244:QuorumCnxManager$Listener@661] - Leaving listener
2016-07-21 04:24:40,897 [myid:] - INFO [WorkerReceiver[myid=8]:QuorumHierarchical@136] - 9, 9, 3
2016-07-21 04:24:40,898 [myid:] - WARN [RecvWorker:7:QuorumCnxManager$RecvWorker@915] - Connection broken for id 7, my id = 8, error =
java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:392)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:900)
2016-07-21 04:24:47,925 [myid:] - WARN [RecvWorker:7:QuorumCnxManager$RecvWorker@918] - Interrupting SendWorker
2016-07-21 04:24:40,898 [myid:] - WARN [RecvWorker:8:QuorumCnxManager$RecvWorker@915] - Connection broken for id 8, my id = 7, error =
java.net.SocketException: Socket closed
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at java.net.SocketInputStream.read(SocketInputStream.java:210)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:900)
2016-07-21 04:24:47,926 [myid:] - WARN [RecvWorker:8:QuorumCnxManager$RecvWorker@918] - Interrupting SendWorker
2016-07-21 04:24:40,898 [myid:] - INFO [main:QuorumBase@398] - Shutting down leader election QuorumPeer
2016-07-21 04:24:43,790 [myid:] - INFO [WorkerSender[myid=3]:FastLeaderElection$Messenger$WorkerSender@470] - WorkerSender is down
2016-07-21 04:24:43,790 [myid:] - INFO [WorkerReceiver[myid=3]:FastLeaderElection$Messenger$WorkerReceiver@440] - WorkerReceiver is down
2016-07-21 04:24:43,642 [myid:] - INFO [WorkerSender[myid=1]:FastLeaderElection$Messenger$WorkerSender@470] - WorkerSender is down
2016-07-21 04:24:43,642 [myid:] - INFO [WorkerReceiver[myid=1]:FastLeaderElection$Messenger$WorkerReceiver@440] - WorkerReceiver is down
2016-07-21 04:24:43,585 [myid:] - INFO [WorkerSender[myid=4]:FastLeaderElection$Messenger$WorkerSender@470] - WorkerSender is down
2016-07-21 04:24:43,574 [myid:] - INFO [WorkerSender[myid=6]:FastLeaderElection$Messenger$WorkerSender@470] - WorkerSender is down
2016-07-21 04:24:40,898 [myid:] - WARN [SendWorker:8:QuorumCnxManager$SendWorker@837] - Exception when using channel: for id 8 my id = 7 error = java.net.SocketException: Socket closed
2016-07-21 04:24:40,898 [myid:] - INFO [WorkerReceiver[myid=7]:QuorumHierarchical@136] - 9, 9, 3
2016-07-21 04:24:40,898 [myid:] - INFO [WorkerReceiver[myid=8]:FastLeaderElection@688] - Notification: 2 (message format version), 4 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 4 (n.sid), 0x0 (n.peerEPoch), FOLLOWING (my state)0 (n.config version)
2016-07-21 04:24:47,927 [myid:] - WARN [SendWorker:8:QuorumCnxManager$SendWorker@841] - Send worker leaving thread id 8 my id = 7
2016-07-21 04:24:47,926 [myid:] - INFO [main:QuorumBase@403] - Waiting for QuorumPeer to exit thread
2016-07-21 04:24:47,926 [myid:] - WARN [SendWorker:7:QuorumCnxManager$SendWorker@832] - Interrupted while waiting for message on queue
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:982)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:63)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:820)
2016-07-21 04:24:47,928 [myid:] - INFO [/127.0.0.1:11247:QuorumCnxManager$Listener@638] - Received connection request /127.0.0.1:57823
2016-07-21 04:24:47,928 [myid:] - INFO [WorkerSender[myid=7]:QuorumCnxManager@276] - Have smaller server identifier, so dropping the connection: (8, 7)
2016-07-21 04:24:47,928 [myid:] - INFO [WorkerReceiver[myid=8]:QuorumHierarchical@136] - 9, 9, 3
2016-07-21 04:24:47,928 [myid:] - WARN [SendWorker:7:QuorumCnxManager$SendWorker@841] - Send worker leaving thread id 7 my id = 8
2016-07-21 04:24:47,929 [myid:] - INFO [main:QuorumBase@394] - Shutting down quorum peer QuorumPeer
2016-07-21 04:24:47,929 [myid:] - WARN [WorkerSender[myid=8]:QuorumCnxManager@455] - Cannot open channel to 4 at election address /127.0.0.1:11235
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:441)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:482)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:419)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:486)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:465)
at java.lang.Thread.run(Thread.java:745)
2016-07-21 04:24:47,929 [myid:] - INFO [WorkerSender[myid=7]:FastLeaderElection$Messenger$WorkerSender@470] - WorkerSender is down
2016-07-21 04:24:47,929 [myid:] - INFO [WorkerReceiver[myid=7]:FastLeaderElection@688] - Notification: 2 (message format version), 2 (n.leader), 0x0 (n.zxid), 0xffffffffffffffff (n.round), FOLLOWING (n.state), 3 (n.sid), 0x0 (n.peerEPoch), FOLLOWING (my state)0 (n.config version)
2016-07-21 04:24:47,930 [myid:] - INFO [WorkerReceiver[myid=7]:FastLeaderElection$Messenger$WorkerReceiver@440] - WorkerReceiver is down
2016-07-21 04:24:47,930 [myid:] - WARN [/127.0.0.1:11247:QuorumCnxManager@455] - Cannot open channel to 7 at election address /127.0.0.1:11244
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:441)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.receiveConnection(QuorumCnxManager.java:367)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$Listener.run(QuorumCnxManager.java:640)
2016-07-21 04:24:47,930 [myid:] - INFO [/127.0.0.1:11247:QuorumCnxManager$Listener@661] - Leaving listener
2016-07-21 04:24:47,930 [myid:] - INFO [WorkerReceiver[myid=8]:FastLeaderElection@688] - Notification: 2 (message format version), 7 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 7 (n.sid), 0x0 (n.peerEPoch), FOLLOWING (my state)0 (n.config version)
2016-07-21 04:24:47,931 [myid:] - INFO [main:QuorumBase@398] - Shutting down leader election QuorumPeer
2016-07-21 04:24:47,931 [myid:] - INFO [main:QuorumBase@403] - Waiting for QuorumPeer to exit thread
2016-07-21 04:24:47,931 [myid:] - INFO [WorkerReceiver[myid=8]:FastLeaderElection$Messenger$WorkerReceiver@440] - WorkerReceiver is down
2016-07-21 04:24:47,932 [myid:] - WARN [WorkerSender[myid=8]:QuorumCnxManager@455] - Cannot open channel to 7 at election address /127.0.0.1:11244
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:441)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:482)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:419)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:486)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:465)
at java.lang.Thread.run(Thread.java:745)
2016-07-21 04:24:47,932 [myid:] - INFO [WorkerSender[myid=8]:FastLeaderElection$Messenger$WorkerSender@470] - WorkerSender is down
2016-07-21 04:24:47,933 [myid:] - INFO [main:ZKTestCase$1@70] - FAILED testZeroWeightQuorum
java.lang.AssertionError: Threads didn't join
at org.junit.Assert.fail(Assert.java:88)
at org.apache.zookeeper.test.FLEZeroWeightTest.testZeroWeightQuorum(FLEZeroWeightTest.java:167)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:79)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.TestWatchman$1.evaluate(TestWatchman.java:53)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:38)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:518)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1052)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:906)
2016-07-21 04:24:47,933 [myid:] - INFO [main:ZKTestCase$1@60] - FINISHED testZeroWeightQuorum
{noformat}
flaky, flaky-test 9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
1 year, 21 weeks ago 0|i31cin:
ZooKeeper ZOOKEEPER-2480

Exhibitor and zookeeper, Cannot open channel to x at election address

Bug Resolved Minor Not A Problem Unassigned Kerem Yazici Kerem Yazici 19/Jul/16 07:11   19/Jul/16 11:45 19/Jul/16 11:45 3.4.8   quorum   0 1   We have 5 node ensemble setup in our tst environment and we are seeing the error below on two of the nodes. We have 3 nodes on the first data centre and 2 nodes on the second data centre and all of them are managed by exhbitor.

The problem is that the nodes from the same data centres cannot talk to the leader node on the same data centre but can talk to the other nodes without any issue.

If I bounce the leader node to force another node to get elected then the other nodes that are on the same data centre start throwing the below exception.

I'm sure the problem is with the dns name resolution but I would like to understand how zookeeper resolves these dns names and what might be the issue here so I can go back to our unix team and get this fixed.


{code}
2016-07-19 10:48:54,711 [myid:4] - WARN [WorkerSender[myid=4]:QuorumCnxManager@400] - Cannot open channel to 5 at election address server1.dns.name/192.168.1.3:4882
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:381)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:354)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(Fast LeaderElection.java:452)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLead erElection.java:433)
at java.lang.Thread.run(Thread.java:745)

{code}



{code:title=zoo.cfg|borderStyle=solid}
#Auto-generated by Exhibitor - Fri Jul 15 11:30:52 BST 2016
#Fri Jul 15 11:30:52 BST 2016
server.2=server2.dns.name\:4881\:4882\:observer
autopurge.purgeInterval=4
server.1=server1.dns.name\:4881\:4882
initLimit=50
syncLimit=2
clientPort=4880
tickTime=2001
server.5=server5.dns.name\:4881\:4882
dataDir=/opt/app/datafabric/data/zookeeper
server.4=server4.dns.name\:4881\:4882
dataLogDir=/path/to/datalogdir
server.3=server3.dns.name\:4881\:4882
~
{code}
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 35 weeks, 2 days ago 0|i316yv:
ZooKeeper ZOOKEEPER-2479

Add 'electionTimeTaken' value in LeaderMXBean and FollowerMXBean

Improvement Closed Major Fixed Rakesh Radhakrishnan Rakesh Radhakrishnan Rakesh Radhakrishnan 19/Jul/16 01:01   31/Mar/17 05:01 20/Dec/16 08:53   3.4.10, 3.5.3, 3.6.0 quorum   0 6   The idea of this jira is to expose {{time taken}} for the leader election via jmx Leader, Follower beans. 9223372036854775807 No Perforce job exists for this issue. 6 9223372036854775807
3 years, 13 weeks ago 0|i316i7:
ZooKeeper ZOOKEEPER-2478

Duplicate 'the' on Zab in words Wiki page

Bug Resolved Trivial Fixed Flavio Paiva Junqueira Richard Shaw Richard Shaw 18/Jul/16 21:12   31/Jan/19 08:48 19/Jul/16 01:02     documentation   0 4   There's a duplicate 'the' on the Zookeeper wiki
Zab in words

Phase 1: Establish an epoch
4.1
documentation 9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
1 year, 7 weeks ago https://cwiki.apache.org/confluence/display/ZOOKEEPER/Zab1.0 0|i316c7:
ZooKeeper ZOOKEEPER-2477

documentation should refer to Java cli shell and not C cli shell

Bug Closed Major Fixed Abraham Fine Patrick D. Hunt Patrick D. Hunt 17/Jul/16 13:09   04/Sep/16 01:28 29/Jul/16 18:02 3.4.8, 3.5.2 3.4.9, 3.5.3, 3.6.0 documentation   0 5   The documentation tends to refer to the c cli shell when citing examples of how to interact with ZK, rather than using the Java cli shell. Given the Java cli is much better maintained and more featureful the docs should refer to that instead. Also the c cli was originally meant to be a sample/example of c client usage rather than a true cli tool. newbie 9223372036854775807 No Perforce job exists for this issue. 6 9223372036854775807
3 years, 33 weeks, 6 days ago
Reviewed
0|i313qv:
ZooKeeper ZOOKEEPER-2476

Not possible to upgrade via reconfig a Participant+Observer cluster to a Participant+Participant cluster

Bug Resolved Critical Not A Bug Alexander Shraer Jordan Zimmerman Jordan Zimmerman 15/Jul/16 20:25   16/Jul/16 20:21 16/Jul/16 20:21 3.5.1   quorum, server   0 2   Contrary to the documentation, it is not possible to upgrade via reconfig a Participant+Observer cluster to a Participant+Participant cluster. KeeperException.NewConfigNoQuorum is thrown instead.

PrepRequestProcessor should recognize this special case and let it pass. Test will be enclosed shortly. I'll work on a fix as well, but I imagine that [~shralex] will want to look at it.
9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
Important
3 years, 35 weeks, 5 days ago 0|i312g7:
ZooKeeper ZOOKEEPER-2475

Include ZKClientConfig API in zoookeeper javadoc

Bug Patch Available Major Unresolved Mohammad Arshad Mohammad Arshad Mohammad Arshad 14/Jul/16 10:06   05/Feb/20 07:11   3.5.2 3.7.0, 3.5.8 build, documentation   0 3   Generate zookeeper api doc using {{ant javadoc}} command
open build/docs/api/index.html, ZKClientConfig is not present
9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
3 years, 32 weeks, 2 days ago 0|i30z8f:
ZooKeeper ZOOKEEPER-2474

add a way for client to reattach to a session when using ZKClientConfig

Bug Closed Major Fixed maoling Timothy James Ward Timothy James Ward 14/Jul/16 08:52   20/May/19 13:50 12/Mar/19 14:09 3.5.2 3.6.0, 3.5.5 java client   0 4 0 14400   The new constructors for ZooKeeper instances take a ZKClientConfig, which is great, however there is no way to reattach to an existing session.

New constructors should be added to allow passing a session id and password when using ZKClientConfig.

 
100% 100% 14400 0 pull-request-available 9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
1 year, 1 week, 2 days ago 0|i30z2n:
ZooKeeper ZOOKEEPER-2473

ZooInspector support for read-only mode

Improvement Open Minor Unresolved Benjamin Jaton Benjamin Jaton Benjamin Jaton 11/Jul/16 13:28   11/Jul/16 13:36   3.5.1   contrib   0 1   9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
3 years, 36 weeks, 3 days ago 0|i30svz:
ZooKeeper ZOOKEEPER-2472

ServerConfig#parse(String[]) parse params has problems

Bug Resolved Minor Duplicate Unassigned yangkun yangkun 10/Jul/16 22:31   10/Jul/16 23:43 10/Jul/16 23:18     server   0 2   When debug zookeeper, run ZooKeeperServerMain then pass 4 args, e.g: 2181 F:\\zk\\data 2000 30, that is:

clientPort = 2181
dataDir = F:\\zk\\data
tickTime = 2000
maxClientCnxns = 30
But ServerConfig#parse(String[]) method has a little problem:

public void parse(String[] args) {
...
if (args.length == 3) {
tickTime = Integer.parseInt(args[2]);
}
if (args.length == 4) {
maxClientCnxns = Integer.parseInt(args[3]);
}
}
The problem is:

if (args.length == 4) {
maxClientCnxns = Integer.parseInt(args[3]);
}
It can't parse tickTime, igone the tickTime.This coe snippet should be:

if (args.length == 4) {
tickTime = Integer.parseInt(args[2]);
maxClientCnxns = Integer.parseInt(args[3]);
}
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 36 weeks, 3 days ago 0|i30rq7:
ZooKeeper ZOOKEEPER-2471

Java Zookeeper Client incorrectly considers time spent sleeping as time spent connecting, potentially resulting in infinite reconnect loop

Bug In Progress Major Unresolved Michael Han Dan Benediktson Dan Benediktson 08/Jul/16 11:52   12/Dec/19 20:26   3.5.3   java client   0 8 0 3000   all ClientCnxnSocket uses a member variable "now" to track the current time, and lastSend / lastHeard variables to track socket liveness. Implementations, and even ClientCnxn itself, are expected to call both updateNow() to reset "now" to System.currentTimeMillis, and then call updateLastSend()/updateLastHeard() on IO completions.

This is a fragile contract, so it's not surprising that there's a bug resulting from it: ClientCnxn.SendThread.run() calls updateLastSendAndHeard() as soon as startConnect() returns, but it does not call updateNow() first. I expect when this was written, either the expectation was that startConnect() was an asynchronous operation and that updateNow() would have been called very recently, or simply the requirement to call updateNow() was forgotten at this point. As far as I can see, this bug has been present since the "updateNow" method was first introduced in the distant past. As it turns out, since startConnect() calls HostProvider.next(), which can sleep, quite a lot of time can pass, leaving a big gap between "now" and now.

If you are using very short session timeouts (one of our ZK ensembles has many clients using a 1-second timeout), this is potentially disastrous, because the sleep time may exceed the connection timeout itself, which can potentially result in the Java client being stuck in a perpetual reconnect loop. The exact code path it goes through in this case is complicated, because there has to be a previously-closed socket still waiting in the selector (otherwise, the first timeout evaluation will not fail because "now" still hasn't been updated, and then the actual connect timeout will be applied in ClientCnxnSocket.doTransport()) so that select() will harvest the IO from the previous socket and updateNow(), resulting in the next loop through ClientCnxnSocket.SendThread.run() observing the spurious timeout and failing. In practice it does happen to us fairly frequently; we only got to the bottom of the bug yesterday. Worse, when it does happen, the Zookeeper client object is rendered unusable: it's stuck in a perpetual reconnect loop where it keeps sleeping, opening a socket, and immediately closing it.

I have a patch. Rather than calling updateNow() right after startConnect(), my fix is to remove the "now" member variable and the updateNow() method entirely, and to instead just call System.currentTimeMillis() whenever time needs to be evaluated. I realize there is a benefit (aside from a trivial micro-optimization not worth worrying about) to having the time be "fixed", particularly for truth in the logging: if time is fixed by an updateNow() call, then the log for a timeout will still show exactly the same value the code reasoned about. However, this benefit is in my opinion not enough to merit the fragility of the contract which led to this (for us) highly impactful and difficult-to-find bug in the first place.

I'm currently running ant tests locally against my patch on trunk, and then I'll upload it here.
100% 100% 3000 0 pull-request-available 9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
1 year, 3 weeks ago 0|i30pkn:
ZooKeeper ZOOKEEPER-2470

ServerConfig#parse(String[]) ignores tickTime

Bug Closed Trivial Fixed Edward Ribeiro Alexander Shraer Alexander Shraer 08/Jul/16 00:59   31/Mar/17 05:01 21/Dec/16 22:12 3.4.7, 3.5.1 3.4.10, 3.5.3, 3.6.0 server   0 8   Based on bug report from ykgarfield:

ServerConfig#parse(String[]) method has the following code:

public void parse(String[] args) {
...
if (args.length == 3) {
tickTime = Integer.parseInt(args[2]);
}
if (args.length == 4) {
maxClientCnxns = Integer.parseInt(args[3]);
}
}
```

So if args.length == 4 tickTime isn't parsed.
9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
3 years, 12 weeks, 6 days ago 0|i30osn:
ZooKeeper ZOOKEEPER-2469

infinite loop in ZK re-login

Bug Patch Available Major Unresolved Sergey Shelukhin Sergey Shelukhin Sergey Shelukhin 07/Jul/16 20:55   05/Aug/16 15:06           0 9   {noformat}
int retry = 1;
while (retry >= 0) {
try {
reLogin();
break;
} catch (LoginException le) {
if (retry > 0) {
--retry;
// sleep for 10 seconds.
try {
Thread.sleep(10 * 1000);
} catch (InterruptedException e) {
LOG.error("Interrupted during login retry after LoginException:", le);
throw le;
}
} else {
LOG.error("Could not refresh TGT for principal: " + principal + ".", le);
}
}
}
{noformat}
will retry forever. Should return like the one above
9223372036854775807 No Perforce job exists for this issue. 4 9223372036854775807
3 years, 32 weeks, 6 days ago 0|i30ojj:
ZooKeeper ZOOKEEPER-2468

SetQuota and DelQuota

Bug Patch Available Major Unresolved SREENIVASULUDANDU Joshi Shankar Joshi Shankar 07/Jul/16 01:34   12/Dec/16 03:39   3.4.6, 3.4.8, 3.5.1   java client   1 5   Windows /Linux Setquota and delquota commands does not work
Steps to reproduce:
1. Create Test Node 1
create -c /TestZookeeperNodeNumber "testdata"

2. Create Test Node 2
create -c /TestZookeeperNodeBytes "testdatabytes"

3. Set Quota Using
setquota -n 1 /TestZookeeperNodeNumber

4. Set Quota Using
setquota -b 10 /TestZookeeperNodeBytes
AlreadySelectedException is thrown by Apache CLI. It is bug in Apache CLI (https://issues.apache.org/jira/browse/CLI-183)

We can fix by upgrading Apache CLI From(*commons-cli-1.2.jar*) to (*commons-cli-1.3.1.jar*)
Client Operation Log:


[zk: localhost:2181(CONNECTED) 2] create -c /TestZookeeperNodeNumber "testdata"
Created /TestZookeeperNodeNumber
[zk: localhost:2181(CONNECTED) 3] create -c /TestZookeeperNodeBytes "testdatabytes"
Created /TestZookeeperNodeBytes
[zk: localhost:2181(CONNECTED) 4] setquota -n 1 /TestZookeeperNodeNumber
[zk: localhost:2181(CONNECTED) 5] setquota -b 10 /TestZookeeperNodeBytes
The option 'b' was specified but an option from this group has already been selected: 'n'
ZooKeeper -server host:port cmd args
addauth scheme auth
close
config [-c] [-w] [-s]
connect host:port
create [-s] [-e] [-c] path [data] [acl]
delete [-v version] path
deleteall path
delquota [-n|-b] path
get [-s] [-w] path
getAcl [-s] path
history
listquota path
ls [-s] [-w] path
ls2 path [watch]
printwatches on|off
quit
reconfig [-s] [-v version] [[-file path] | [-members serverID=host:port1:port2;port3[,...]*]
] | [-add serverId=host:port1:port2;port3[,...]]* [-remove serverId[,...]*]
redo cmdno
removewatches path [-c|-d|-a] [-l]
rmr path
set [-s] [-v version] path data
setAcl [-s] [-v version] path acl
setquota -n|-b val path
stat [-w] path
sync path
9223372036854775807 No Perforce job exists for this issue. 2 9223372036854775807
3 years, 14 weeks, 3 days ago https://issues.apache.org/jira/browse/CLI-183 0|i30mun:
ZooKeeper ZOOKEEPER-2467

NullPointerException when redo Command is passed negative value

Bug Closed Minor Fixed Rakesh Kumar Singh Joshi Shankar Joshi Shankar 06/Jul/16 22:51   31/Mar/17 05:01 16/Oct/16 09:18 3.4.8, 3.5.1, 3.5.2 3.4.10, 3.5.3, 3.6.0 java client   0 6   Linux , windows When negative value of argument is passed to redo command .

[zk: localhost:2181(CONNECTED) 0] redo -1
Exception in thread "main" java.lang.NullPointerException
at java.util.StringTokenizer.<init>(Unknown Source)
at java.util.StringTokenizer.<init>(Unknown Source)
at org.apache.zookeeper.ZooKeeperMain$MyCommandOptions.parseCommand(ZooKeeperMain.java:227)
at org.apache.zookeeper.ZooKeeperMain.processZKCmd(ZooKeeperMain.java:645)
at org.apache.zookeeper.ZooKeeperMain.processCmd(ZooKeeperMain.java:588)
at org.apache.zookeeper.ZooKeeperMain.executeLine(ZooKeeperMain.java:360)
at org.apache.zookeeper.ZooKeeperMain.run(ZooKeeperMain.java:323)
at org.apache.zookeeper.ZooKeeperMain.main(ZooKeeperMain.java:282)
9223372036854775807 No Perforce job exists for this issue. 8 9223372036854775807
3 years, 22 weeks, 4 days ago 0|i30ms7:
ZooKeeper ZOOKEEPER-2466

Client skips servers when trying to connect

Bug Patch Available Major Unresolved Michael Han Flavio Paiva Junqueira Flavio Paiva Junqueira 05/Jul/16 17:04   14/Dec/19 06:07     3.7.0 c client   0 7   I've been looking at {{Zookeeper_simpleSystem::testFirstServerDown}} and I observed the following behavior. The list of servers to connect contains two servers, let's call them S1 and S2. The client never connects, but the odd bit is the sequence of servers that the client tries to connect to:

{noformat}
S1
S2
S1
S1
S1
<keeps repeating S1>
{noformat}

It intrigued me that S2 is only tried once and never again. Checking the code, here is what happens. Initially, {{zh->reconfig}} is 1, so in {{zoo_cycle_next_server}} we return an address from {{get_next_server_in_reconfig}}, which is taken from {{zh->addrs_new}} in this test case. The attempt to connect fails, and {{handle_error}} is invoked in the error handling path. {{handle_error}} actually invokes {{addrvec_next}} which changes the address pointer to the next server on the list.

After two attempts, it decides that it has tried all servers in {{zoo_cycle_next_server}} and sets {{zh->reconfig}} to zero. Once {{zh->reconfig == 0}}, we have that each call to {{zoo_cycle_next_server}} moves the address pointer to the next server in {{zh->addrs}}. But, given that {{handle_error}} also moves the pointer to the next server, we end up moving the pointer ahead twice upon every failed attempt to connect, which is wrong.
9223372036854775807 No Perforce job exists for this issue. 2 9223372036854775807
1 year, 17 weeks ago 0|i30k7z:
ZooKeeper ZOOKEEPER-2465

Documentation copyright notice is out of date.

Bug Closed Blocker Fixed Edward Ribeiro Chris Nauroth Chris Nauroth 05/Jul/16 13:38   31/Mar/17 05:01 30/Dec/16 17:59   3.4.10, 3.5.3, 3.6.0 documentation   0 8   As reported by [~eribeiro], all of the documentation pages show a copyright notice dating "2008-2013". This issue tracks updating the copyright notice on all documentation pages to show the current year. 9223372036854775807 No Perforce job exists for this issue. 2 9223372036854775807
3 years, 11 weeks, 6 days ago 0|i30jsv:
ZooKeeper ZOOKEEPER-2464

NullPointerException on ContainerManager

Bug Closed Major Fixed Jordan Zimmerman Stefano Salmaso Stefano Salmaso 04/Jul/16 18:21   17/May/17 23:43 11/Feb/17 10:09 3.5.1 3.5.3, 3.6.0 server   2 8   I would like to expose you to a problem that we are experiencing.
We are using a cluster of 7 zookeeper and we use them to implement a distributed lock using Curator (http://curator.apache.org/curator-recipes/shared-reentrant-lock.html)
So .. we tried to play with the servers to see if everything worked properly and we stopped and start servers to see that the system worked well
(like stop 03, stop 05, stop 06, start 05, start 06, start 03)

We saw a strange behavior.
The number of znodes grew up without stopping (normally we had 4000 or 5000, we got to 60,000 and then we stopped our application)

In zookeeeper logs I saw this (on leader only, one every minute)

2016-07-04 14:53:50,302 [myid:7] - ERROR [ContainerManagerTask:ContainerManager$1@84] - Error checking containers
java.lang.NullPointerException
at org.apache.zookeeper.server.ContainerManager.getCandidates(ContainerManager.java:151)
at org.apache.zookeeper.server.ContainerManager.checkContainers(ContainerManager.java:111)
at org.apache.zookeeper.server.ContainerManager$1.run(ContainerManager.java:78)
at java.util.TimerThread.mainLoop(Timer.java:555)
at java.util.TimerThread.run(Timer.java:505)

We have not yet deleted the data ... so the problem can be reproduced on our servers
9223372036854775807 No Perforce job exists for this issue. 2 9223372036854775807
3 years, 5 weeks, 5 days ago 0|i30ijr:
ZooKeeper ZOOKEEPER-2463

TestMulti is broken in the C client

Bug Closed Blocker Cannot Reproduce Unassigned Flavio Paiva Junqueira Flavio Paiva Junqueira 03/Jul/16 11:38   19/Dec/19 18:01 24/Nov/16 11:31   3.5.3     0 2   I noticed that all multi tests seem to be timing out and they are failing silently. This is the output that I'm observing:

{noformat}
Zookeeper_multi::testCreate : assertion : elapsed 10001
Zookeeper_multi::testCreateDelete : assertion : elapsed 10001
Zookeeper_multi::testInvalidVersion : assertion : elapsed 10001
Zookeeper_multi::testNestedCreate : assertion : elapsed 10001
Zookeeper_multi::testSetData : assertion : elapsed 10001
Zookeeper_multi::testUpdateConflict : assertion : elapsed 10001
Zookeeper_multi::testDeleteUpdateConflict : assertion : elapsed 10001
Zookeeper_multi::testAsyncMulti : assertion : elapsed 10001
Zookeeper_multi::testMultiFail : assertion : elapsed 10001
Zookeeper_multi::testCheck : assertion : elapsed 10001
Zookeeper_multi::testWatch : assertion : elapsed 10001
Zookeeper_multi::testSequentialNodeCreateInAsyncMulti : assertion : elapsed 10001
{noformat}
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 17 weeks ago 0|i30h9r:
ZooKeeper ZOOKEEPER-2462

force authentication/authorization

New Feature Open Minor Unresolved Unassigned Botond Hejj Botond Hejj 01/Jul/16 10:50   04/Oct/19 10:55       server   1 4   This change introduces two new config options to force authorization and authentication:

1. disableWorldACL
The purpose of this option is disable the builtin mechanism which authorizes everyone.
If it is turned on than the world/anyone usage is ignored. ZooKeeper will not check operations based on world/anyone.
This option is useful to force some kind of authorization mechanism. This restriction is useful in a strictly audited environment.

2. forceAuthentication
If this option is turned on than ZooKeeper won't authorize any operation if the user has not authenticated either with SASL or with addAuth.
There is way to enforce SASL authentication but currently there is no way to enforce authentication using the plugin mechanism. Enforcing authentication for that is more tricky since authentication can come any time later. This option doesn't drop the connection if there was no authentication. It is only throwing NoAuth for any operation until the Auth packet arrives.

9223372036854775807 No Perforce job exists for this issue. 2 9223372036854775807
1 year, 36 weeks, 2 days ago 0|i30fn3:
ZooKeeper ZOOKEEPER-2461

There is no difference between the observer and the participants in the leader election algorithm

Improvement Open Major Unresolved Ryan Zhang Ryan Zhang Ryan Zhang 30/Jun/16 13:59   05/Feb/20 07:16   3.5.0 3.7.0, 3.5.8 quorum   0 5   We have observed a case that when a leader machine crashes hard, non-voting learners take a long time to detect the new leader. After looking at the details more carefully, we identified one potential improvement (and one bug fixed in the 3.5). 9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
Patch
1 year, 14 weeks, 1 day ago 0|i30dt3:
ZooKeeper ZOOKEEPER-2460

Remove javacc dependency from public Maven pom

Bug Closed Critical Fixed Enrico Olivelli Enrico Olivelli Enrico Olivelli 30/Jun/16 03:39   17/May/17 23:49 16/Mar/17 14:09 3.5.2 3.5.3, 3.6.0 java client   1 7   during the vote of 3.5.2-ALPHA RC 0 we found a Maven dependency to javacc in published pom for zookeeper

{code}
<dependency>
<groupId>net.java.dev.javacc</groupId>
<artifactId>javacc</artifactId>
<version>5.0</version><scope>compile</scope>
</dependency>
{code}

this dependency appears not to be useful and should be removed

this was the tested pom: https://repository.apache.org/content/groups/staging/org/apache/zookeeper/zookeeper/3.5.2-alpha/zookeeper-3.5.2-alpha.pom
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 1 week ago 0|i30cqf:
ZooKeeper ZOOKEEPER-2459

Update NOTICE file with Netty notice

Bug Closed Blocker Fixed Flavio Paiva Junqueira Flavio Paiva Junqueira Flavio Paiva Junqueira 29/Jun/16 18:43   21/Jul/16 16:18 30/Jun/16 00:13   3.5.2, 3.6.0     0 4   Bubbling up the Netty notice. According to the ALv2 item 4, we need to include it in our top notice, it isn't sufficient to have it in the bundle. 9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
3 years, 38 weeks ago
Reviewed
0|i30c7z:
ZooKeeper ZOOKEEPER-2458

Remove license file for servlet-api dependency

Bug Closed Major Fixed Flavio Paiva Junqueira Flavio Paiva Junqueira Flavio Paiva Junqueira 29/Jun/16 17:48   21/Jul/16 16:18 29/Jun/16 23:59 3.5.2, 3.6.0 3.5.2, 3.6.0     0 3   In ZOOKEEPER-2235, we changed the license of the servlet-api dependency to the correct one ALv2, but didn't remove the CDDL license file, which is incorrect. This jira removes the incorrect license file. 9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
3 years, 38 weeks ago
Reviewed
0|i30c4f:
ZooKeeper ZOOKEEPER-2457

Remove license file for servlet-api dependency

Bug Closed Major Duplicate Flavio Paiva Junqueira Flavio Paiva Junqueira Flavio Paiva Junqueira 29/Jun/16 17:47   19/Dec/19 18:01 29/Jun/16 17:55   3.5.2     0 1   In ZOOKEEPER-2235, we changed the license of the servlet-api dependency to the correct one ALv2, but didn't remove the CDDL license file, which is incorrect. This jira removes the incorrect license file. 9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 38 weeks, 1 day ago 0|i30c3z:
ZooKeeper ZOOKEEPER-2456

Provide API to get user from different authentication providers

Improvement Open Major Unresolved Mohammad Arshad Mohammad Arshad Mohammad Arshad 29/Jun/16 15:32   05/Feb/20 07:16     3.7.0, 3.5.8 server   0 2   Currently zookeeper server same field is used to store both user name and password
Provide a mechanism to separate the user and password either by adding new field or by adding new API

DETAILS:
org.apache.zookeeper.data.Id class is used to store scheme and id.
{code}
public Id( String scheme, String id)
{code}
id field holds only user in most cases but in some cases it holds user as well as password
By default there are only four authentication provider
DigestAuthenticationProvider
IPAuthenticationProvider
SASLAuthenticationProvider
X509AuthenticationProvider
In code we can check if scheme is digest then {{id.split(":")\[0\]}} is user otherwise id is user. This will work only if we are limited to above four authentication provider
But Custom authentication provider are very important and are very commonly used. How the zookeeper code will know what is the user, is it id or {{id.split(":")\[0\]}} or anything else ?
So there is need to add new API which AuthenticationProvider providers implement to define what is user.
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 38 weeks, 1 day ago 0|i30c13:
ZooKeeper ZOOKEEPER-2455

unexpected server response ZRUNTIMEINCONSISTENCY

Bug Open Major Unresolved Unassigned pradeep pradeep 28/Jun/16 02:01   05/Feb/20 07:17   3.5.1 3.7.0, 3.5.8 c client   0 6   Hi Folks,

I am hitting an error in my C client code and below are the set of operations I perform:

1. Zookeeper Client connected to Zookeeper server S1 and a new server S2 gets added.
2. monitor zookeeper server config at the client and on change of server config, call zoo_set_server
from the client
3. client can issue operations like zoo_get just after the call to zoo_set_servers
4. I can see that the zookeeper thread logs connect to the new server just after the zoo_get
call

2016-04-11 03:46:50,655:1207(0xf26ffb40):ZOO_INFO@check_events@2345: initiated connection
to server [128.0.0.5:61728]

2016-04-11 03:46:50,658:1207(0xf26ffb40):ZOO_INFO@check_events@2397: session establishment
complete on server [128.0.0.5:61728], sessionId=0x4000001852c000c, negotiated timeout=20000

5. Some times I find errors like below:

2016-04-11 03:46:50,662:1207(0xf26ffb40):ZOO_ERROR@handle_socket_error_msg@2923: Socket [128.0.0.5:61728]
zk retcode=-2, errno=115(Operation now in progress): unexpected server response: expected
0x570b82fa, but received 0x570b82f9

1.
zoo_get returns (-2) indicating that ZRUNTIMEINCONSISTENCY<http://zookeeper.sourcearchive.com/documentation/3.2.2plus-pdfsg3/zookeeper_8h_bb1a0a179f313b2e44ee92369c438a4c.html#bb1a0a179f313b2e44ee92369c438a4c9eabb281ab14c74db3aff9ab456fa7fe>


What is the issue here? should I be retry the operation zoo_get operation? Or should I wait
for the zoo_set_server to complete (like wait for the connection establishment notification)

Thanks,
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
16 weeks, 5 days ago 0|i308k7:
ZooKeeper ZOOKEEPER-2454

Limit Connection Count based on User

New Feature Open Minor Unresolved Botond Hejj Botond Hejj Botond Hejj 27/Jun/16 05:16   09/Oct/16 21:54       server   0 8   ZooKeeper currently can limit connection count from clients coming from the same ip. It is a great feature to malfunctioning clients DOS-ing the server with many requests.

I propose additional safegurads for ZooKeeper.
It would be great if optionally connection count could be limited for a specific user or a specific user on an ip.
This is great in cases where ZooKeeper ensemble is shared by multiple users and these users share the same client ips. This can be common in container based cloud deployment where external ip of multiple clients can be the same.
9223372036854775807 No Perforce job exists for this issue. 3 9223372036854775807
3 years, 23 weeks, 3 days ago 0|i304jb:
ZooKeeper ZOOKEEPER-2453

Cannot compile on ARM: "Error: bad instruction `lock xaddl"

Bug Open Minor Unresolved Unassigned Markus Thies Markus Thies 25/Jun/16 12:49   28/Jun/16 22:51   3.4.5   c client   1 7   Jessie, Raspberry It seems that this is a bug equivalent to the issue ZOOKEEPER-1374.

make[5]: Entering directory '/home/pi/Downloads/mesos-0.28.2/build/3rdparty/zookeeper-3.4.5/src/c'
if /bin/bash ./libtool --tag=CC --mode=compile gcc -DHAVE_CONFIG_H -I. -I. -I. -I./include -I./tests -I./generated -DTHREADED -g -O2 -D_GNU_SOURCE -MT libzkmt_la-mt_adaptor.lo -MD -MP -MF ".deps/libzkmt_la-mt_adaptor.Tpo" -c -o libzkmt_la-mt_adaptor.lo `test -f 'src/mt_adaptor.c' || echo './'`src/mt_adaptor.c; \
then mv -f ".deps/libzkmt_la-mt_adaptor.Tpo" ".deps/libzkmt_la-mt_adaptor.Plo"; else rm -f ".deps/libzkmt_la-mt_adaptor.Tpo"; exit 1; fi
gcc -DHAVE_CONFIG_H -I. -I. -I. -I./include -I./tests -I./generated -DTHREADED -g -O2 -D_GNU_SOURCE -MT libzkmt_la-mt_adaptor.lo -MD -MP -MF .deps/libzkmt_la-mt_adaptor.Tpo -c src/mt_adaptor.c -fPIC -DPIC -o libzkmt_la-mt_adaptor.o
/tmp/ccs0G1lb.s: Assembler messages:
/tmp/ccs0G1lb.s:1589: Error: bad instruction `lock xaddl r1,[r0]'
Makefile:743: recipe for target 'libzkmt_la-mt_adaptor.lo' failed
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 38 weeks, 1 day ago 0|i303fr:
ZooKeeper ZOOKEEPER-2452

Back-port ZOOKEEPER-1460 to 3.4 for IPv6 literal address support.

Bug Closed Critical Fixed Abraham Fine Chris Nauroth Chris Nauroth 23/Jun/16 16:17   04/Sep/16 01:27 11/Aug/16 12:14   3.4.9 quorum   0 4   Via code inspection, I see that the "server.nnn" configuration key does not support literal IPv6 addresses because the property value is split on ":". In v3.4.3, the problem is in QuorumPeerConfig:

{noformat}
String parts[] = value.split(":");
InetSocketAddress addr = new InetSocketAddress(parts[0],
Integer.parseInt(parts[1]));
{noformat}

In the current trunk (http://svn.apache.org/viewvc/zookeeper/trunk/src/java/main/org/apache/zookeeper/server/quorum/QuorumPeer.java?view=markup) this code has been refactored into QuorumPeer.QuorumServer, but the bug remains:

{noformat}
String serverClientParts[] = addressStr.split(";");
String serverParts[] = serverClientParts[0].split(":");
addr = new InetSocketAddress(serverParts[0],
Integer.parseInt(serverParts[1]));
{noformat}

This bug probably affects very few users because most will naturally use a hostname rather than a literal IP address. But given that IPv6 addresses are supported for clients via ZOOKEEPER-667 it seems that server support should be fixed too.
9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
3 years, 32 weeks ago
Reviewed
0|i2zzrj:
ZooKeeper ZOOKEEPER-2451

do not make use of system properties for security configuration

Bug Resolved Major Duplicate Unassigned Sergey Shelukhin Sergey Shelukhin 23/Jun/16 15:32   23/Jun/16 15:56 23/Jun/16 15:56         0 3   Is there (or could there be) a way to set up security for ZK client that doesn't involve calls like {noformat}
System.setProperty(ZooKeeperSaslClient.LOGIN_CONTEXT_NAME_KEY, SASL_LOGIN_CONTEXT_NAME);
{noformat}?
I was looking at an unrelated security configuration issue and stumbled upon this pattern; we use (at least) 2 ZK connections from the same process, that (for now) use the same config but different context names, one of which is in a library out of our control. Unless I'm missing something with this pattern it seems extremely brittle. Or unless there's an alternative approach already; if there is, hadoop-common and hive don't use it atm, old approach seems prevalent.

There should be an approach that is at least slightly more solid, like say public globals... maybe even threadlocals!

9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 39 weeks ago 0|i2zzov:
ZooKeeper ZOOKEEPER-2450

Upgrade Netty version due to security vulnerability (CVE-2014-3488)

Bug Closed Critical Fixed Michael Han Michael Han Michael Han 22/Jun/16 17:11   21/Jul/16 16:18 22/Jun/16 17:12 3.4.8, 3.5.1, 3.6.0 3.4.9, 3.5.2, 3.6.0 security, server   0 2   This JIRA recreates ZOOKEEPER-2432 which was deleted as the collateral damage during the spamming fighting effort Apache Infrastructure Team did weeks ago. Recreate the JIRA for the record so external documentations can link back to this JIRA.

The SslHandler in Netty before 3.9.2 allows remote attackers to cause a denial of service (infinite loop and CPU consumption) via a crafted SSLv2Hello message [1]. We are using netty 3.7.x in ZK for 3.4/3.5/3.6, which is affected by this vulnerability.

[1] http://cve.mitre.org/cgi-bin/cvename.cgi?name=2014-3488
[2] http://netty.io/news/
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 39 weeks, 1 day ago 0|i2zxlb:
ZooKeeper ZOOKEEPER-2449

Refresh the invalid host which got unknownhostexception during connection in StaticHostProvider

Improvement Open Minor Unresolved Unassigned Vishal Khandelwal Vishal Khandelwal 20/Jun/16 08:52   20/Jun/16 13:07           0 3   As per the current logic if a host has "UnknownHostException" during zookeeper client object connection and it is never refreshed after that. Incase host comes back, this class won't try to connect it back.

Ideally in StaticHostProvider.next we end of the list is reached then all of the host in connection string should be tested and refreshed again. That way incase host comes back and object stays for a longer duration, realtime status can be updated and client can take advantage of that

This also benefit the scenario defined https://issues.apache.org/jira/browse/ZOOKEEPER-2447
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
Patch
3 years, 39 weeks, 3 days ago 0|i2zqn3:
ZooKeeper ZOOKEEPER-2448

Expose result of reconfiguration via JMX

Improvement Open Major Unresolved Unassigned Flavio Paiva Junqueira Flavio Paiva Junqueira 19/Jun/16 11:16   19/Jun/16 11:16   3.5.1       0 2   Currently, the main way to get the result of a reconfiguration is to look at the server logs. One possible way to expose a reconfiguration command and its result is to do it via JMX. One advantage over doing a 4lw is that in the case we mess up with the client port, we can still see what is going on. 9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 39 weeks, 4 days ago 0|i2zpin:
ZooKeeper ZOOKEEPER-2447

Zookeeper adds good delay when one of the quorum host is not reachable

Bug Patch Available Major Unresolved Vishal Khandelwal Vishal Khandelwal Vishal Khandelwal 15/Jun/16 08:56   11/Nov/16 02:30   3.4.9 3.4.9     0 7   StaticHostProvider --> resolveAndShuffle method adds all of the address which are valid in the quorum to the list, shuffles them and sends back to client connection class. If after shuffling if first node appear to be the one which is not reachable, Clientcnx.SendThread.run will keep on connecting to the failure till a timeout and the moves to a different node. This adds up random delay in zookeeper connection in case a host is down. Rather we could check if host is reachable in StaticHostProvider and ignore isReachable is false. Same as we do for UnknownHostException Exception.

This can tested using following test code by providing a valid host which is not reachable. for quick test comment Collections.shuffle(tmpList, sourceOfRandomness); in StaticHostProvider.resolveAndShuffle

{code}
@Test
public void test() throws Exception {
EventsWatcher watcher = new EventsWatcher();
QuorumUtil qu = new QuorumUtil(1);
qu.startAll();

ZooKeeper zk =
new ZooKeeper("<hostnamet:2181," + qu.getConnString(), 180 * 1000, watcher);

watcher.waitForConnected(CONNECTION_TIMEOUT * 5);
Assert.assertTrue("connection Established", watcher.isConnected());
zk.close();
}
{code}

Following fix can be added to StaticHostProvider.resolveAndShuffle
{code}
if(taddr.isReachable(4000 // can be some value)) {
tmpList.add(new InetSocketAddress(taddr, address.getPort()));
}
{code}
9223372036854775807 No Perforce job exists for this issue. 6 9223372036854775807
3 years, 18 weeks, 6 days ago 0|i2zhtz:
ZooKeeper ZOOKEEPER-2446

License missing in the pom.xml

Bug Resolved Critical Duplicate Unassigned Sergio Fernández Sergio Fernández 08/Jun/16 06:03   08/Jun/16 06:32 08/Jun/16 06:32 3.4.8   build   0 2   Assembling a {{NOTICE}} file in a project that uses Zookeeper I've realized the {{pom.xml}} does not declare the license, at least in the whole {{3.4.x}} branch, e.g., https://repo1.maven.org/maven2/org/apache/zookeeper/zookeeper/3.4.8/zookeeper-3.4.8.pom maven 9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 41 weeks, 1 day ago 0|i2z5f3:
ZooKeeper ZOOKEEPER-2445

Add a new node to cluster, error config file but sucessed

Bug Open Critical Unresolved Unassigned Ahaha Ahaha 08/Jun/16 04:56   08/Jun/16 10:33   3.4.6   server   0 2 86400 86400 0% Centos 6.5 JDK7 I want to add a new node to a test cluster with three node. So I tried with the folling steps: 1. I copied one foller zookeeper directory, and edit clientPort, dataDir , logDir, myid file, and then add a new record server.newId=hostname:newPort:newEPort into zoo.cfg, for test I keep the leader configuration there and remove the other two configuration, I tested ,this failed. So I open the zoo.cfg , copied the lead configuration line just edit the serverId, and this should be a wrong configuration file as the server does not match with the serverid. But strangely , it successed. 0% 0% 86400 86400 9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 41 weeks, 1 day ago 0|i2z5an:
ZooKeeper ZOOKEEPER-2444

Fix NULL handling for getlogin() call

Bug Open Major Unresolved Unassigned Pawel Rozlach Pawel Rozlach 07/Jun/16 07:01   09/Jun/16 00:58   3.4.8   c client   0 3   https://github.com/apache/zookeeper/pull/70 9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 41 weeks ago 0|i2z2xz:
ZooKeeper ZOOKEEPER-2443

Invalid Memory Access (SEGFAULT) and undefined behaviour in c client

Bug Open Major Unresolved Chris Nauroth Paul Asmuth Paul Asmuth 07/Jun/16 05:42   15/Jun/16 00:13       c client   1 3   Hey,

I encountered some issues with the zookeepeer c client.

The problem starts in the zookeeper_init_internal method. A lot of initialization work is performed here and if any of the initialization routines fails, the code jumps to the "abort" label to perform various cleanup tasks [1]. The conceptual issue is that a bunch of the cleanup code tries to take locks on the zk structure that are only intialized in adaptor_init in line 1181 (at the very end of the zookeeper_init_internal method) [2]. So if we fail before reaching adaptor_init this causes trouble.

One specific instance of an invalid memory access that this causes is in free_completions [3]. Here, in line 1651 zoo_lock_auth will fail because it tries to grab an invalid mutex, after which the a_list struct is uninitialized (the linked list next pointer points to random memory) and subsequently the free routine segfaults.

An easy way to trigger this bug-path is to pass an invalid hostname, or do anything else that causes the zookeeper_init_internal method to fail before adaptor_init.

In my local checkout/codebase, I have added correct initialization for the a_list struct in the free_completions routine, which at least fixes the segfault for now. However this still leaves the issue that the cleanup code tries to grab a lot of invalid locks, which all fail. I think in order to fix this properly, one would need to do a larger refactoring of the code (add another adaptor_preinit routine to the adaptor interface maybe?) and I wasn't sure if that would be appreciated, so I didn't attach a patch for now. If someone wants me to try and clean this up, I would be happy to give it a try.

PS: I think this bug was introduced in SVN #1719528, which - as it seems - tried to work around the uninitialized locks problem by adding an int return code to all the lock_xxx functions, allowing them to indicate a failure. The change introduce the invalid memory access since some (always required) init code is only run after the lock was obtained successfully.

However, I think there is a much large issue with the change and I think it must be reverted. Trying to lock an uninitialized mutex is undefined behaviour on POSIX and may lead to deadlocks, etc.

>> If mutex does not refer to an initialized mutex object, the behavior of pthread_mutex_lock(), pthread_mutex_trylock(), and pthread_mutex_unlock() is undefined.

http://pubs.opengroup.org/onlinepubs/9699919799/functions/pthread_mutex_lock.html

[1] https://github.com/apache/zookeeper/blob/trunk/src/c/src/zookeeper.c#L1078
[2] https://github.com/apache/zookeeper/blob/trunk/src/c/src/zookeeper.c#L1181
[3] https://github.com/apache/zookeeper/blob/trunk/src/c/src/zookeeper.c#L1651

------------------

BACKTRACE

Program received signal SIGSEGV, Segmentation fault.

0x000000010004f6d5 in free_auth_completion (a_list=0x7fff5fbff048) at /deps/3rdparty/zookeeper/source/src/zookeeper.c:260
260 tmp = tmp->next;

#0 0x000000010004f6d5 in free_auth_completion (a_list=0x7fff5fbff048) at /deps/3rdparty/zookeeper/source/src/zookeeper.c:260
#1 0x000000010004f500 in free_completions (zh=0x1003022f0, callCompletion=1, reason=-116) at /deps/3rdparty/zookeeper/source/src/zookeeper.c:1219
#2 0x0000000100057bfd in cleanup_bufs (zh=0x1003022f0, callCompletion=1, rc=-116) at /deps/3rdparty/zookeeper/source/src/zookeeper.c:1227
#3 0x000000010004ee42 in destroy (zh=0x1003022f0) at /deps/3rdparty/zookeeper/source/src/zookeeper.c:393
#4 0x000000010004eaf3 in zookeeper_init (host=0x1006005b0 "xxxinvalidhostname:2181", watcher=0x100007670 <xxx::zk_watch_cb(_zhandle*, int, int, char const*, void*)>,
recv_timeout=10000, clientid=0x0, context=0x100600350, flags=0) at /deps/3rdparty/zookeeper/source/src/zookeeper.c:877
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 40 weeks, 1 day ago 0|i2z2tj:
ZooKeeper ZOOKEEPER-2442

Socket leak in QuorumCnxManager connectOne

Bug Closed Major Fixed Michael Han Michael Han Michael Han 03/Jun/16 19:05   17/May/17 23:43 03/Aug/16 00:32 3.5.1 3.5.3, 3.6.0 quorum, server   0 7   The function connectOne() in QuorumCnxManager.java sometimes fails to release a socket allocated by Socket():

{code}
try {

if (LOG.isDebugEnabled()) {
LOG.debug("Opening channel to server " + sid);
}
Socket sock = new Socket();
setSockOpts(sock);
sock.connect(self.getView().get(sid).electionAddr, cnxTO);
if (LOG.isDebugEnabled()) {
LOG.debug("Connected to server " + sid);
}
initiateConnection(sock, sid);
} catch (UnresolvedAddressException e) {
// Sun doesn't include the address that causes this
// exception to be thrown, also UAE cannot be wrapped cleanly
// so we log the exception in order to capture this critical
// detail.
LOG.warn("Cannot open channel to " + sid
+ " at election address " + electionAddr, e);
throw e;
} catch (IOException e) {
LOG.warn("Cannot open channel to " + sid
+ " at election address " + electionAddr,
e);
}
{code}

Another place in Listener.run() where the client socket is not explicitly closed.
9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
3 years, 21 weeks, 1 day ago 0|i2yz2v:
ZooKeeper ZOOKEEPER-2441

C API maps getaddrinfo() transient and permanent failures to the same value

Bug Open Major Unresolved Unassigned Andras Erdei Andras Erdei 03/Jun/16 04:37   03/Jun/16 04:37           0 1   Linux https://github.com/apache/zookeeper/blob/trunk/src/c/src/zookeeper.c#L560
maps getaddrinfo() return values indicating a transient failure (e.g. EAI_AGAIN) to same value (EINVAL) that zookeeper_init() uses to indicate permanent problems (like empty host spec or invalid port).
As a result client code has no way to decide whether it should re-try the initialization or abort (asking for manual intervention).

As discussed e.g. in https://issues.apache.org/jira/browse/MESOS-3790 zookeeper should most likely retry on this and other transient failures automagically. Independently, the switch above should be fixed to map EAI_* values to different E* values allowing client code some flexibility in handling and reporting errors deemed permanent by zookeeper.

Note that there is a related bug https://issues.apache.org/jira/browse/ZOOKEEPER-1451 -- zookeeper also does not report the problem properly in its own logs, making debugging these problems even harder.
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 41 weeks, 6 days ago 0|i2yxl3:
ZooKeeper ZOOKEEPER-2440

permanent SESSIONMOVED error after client app reconnects to zookeeper cluster

Bug Open Major Unresolved Ryan Zhang Ryan Zhang Ryan Zhang 02/Jun/16 17:01   05/Feb/20 07:16   3.5.0 3.7.0, 3.5.8 quorum   0 6   ZOOKEEPER-710 fixed the issue when the request is not a multi request. However, the multi request is handled a little bit differently as the code didn't throw the SESSIONMOVED exception. In addition, the exception is set in the request by the leader so it will be lost in the commit process and by the time the final processor sees it, it will be gone. 9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
Patch
1 year, 17 weeks, 1 day ago 0|i2ywun:
ZooKeeper ZOOKEEPER-2439

The order of asynchronous setACL is not correct.

Bug Open Major Unresolved Unassigned Kazuaki Banzai Kazuaki Banzai 01/Jun/16 23:17   19/Sep/16 04:05   3.4.8, 3.5.1       1 4   Linux Ubuntu
Mac OS X
Within a given client connection, the execution of commands on the ZooKeeper server is always ordered, as both synchronous and asynchronous commands are dispatched through queuePacket (directly or indirectly).

In other words, Zookeeper guarantees sequential consistency: updates from a client will be applied in the order that they were sent.

However, the order of asynchronous setACL is not correct on Ubuntu.
When asynchronous setACL is called BEFORE another API is called, asynchronous setACL is applied AFTER another API.

For example, if a client calls

(1) asynchronous setACL to remove all permissions of node "/" and

(2) synchronous create to create node "/a",
synchronous create should fail, but it succeeds on Ubuntu.

(We can see all permissions of node "/" are removed when the client calls getACL to node "/" after (2), so (1) is applied AFTER (2). If we call getACL between (1) and (2), the synchronous case works correctly but the asynchronous case still produces the bug.)

The attached unit test reproduces this scenario. It fails on Linux Ubuntu but succeeds on Mac OS X. If used on a heavily loaded server on Mac OS, the test sometimes fails as well but only rarely.
acl 9223372036854775807 No Perforce job exists for this issue. 2 9223372036854775807
3 years, 26 weeks, 3 days ago 0|i2yv7j:
ZooKeeper ZOOKEEPER-2438

Create sample program for multi client single JVM scenario.

Wish Patch Available Minor Unresolved Mohammad Arshad Mohammad Arshad Mohammad Arshad 01/Jun/16 11:21   01/Jun/16 11:46           0 1   Create sample program for multi client single JVM scenario which is handled in ZOOKEEPER-2139. 9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
3 years, 42 weeks, 1 day ago 0|i2ytzz:
ZooKeeper ZOOKEEPER-2437

Need detailed diagnostics for Zookeeper Connection Issues

Improvement Open Critical Unresolved Unassigned harcor harcor 31/May/16 15:43   02/Jun/16 17:42   3.5.1   java client   0 3   Using a zookeeper ensemble with Apache Solr, the client connection (socket) can be disconnected by either Solr or Zookeeper. If the connection fails on the Zookeeper and we are in DEBUG mode then additional diagnostics should be written to the log withe connection exception. 9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 42 weeks ago 0|i2yraf:
ZooKeeper ZOOKEEPER-2436

Inconsistent truncation logic and out of sync logging and comments in recovery code

Bug Open Minor Unresolved Unassigned Ed Rowe Ed Rowe 30/May/16 04:39   30/May/16 04:40           1 3   Consider this scenario:
# Ensemble of nodes A, B, C, D, E with A as the leader
# Nodes A, B get partitioned from C, D, E
# Leader A receives a write _before it detects_ that it has lost its quorum so it logs the write
# Nodes C, D, E elect node C as the leader
# Partition resolves, and nodes A, B rejoin the C, D, E ensemble with C continuing to lead

Depending on whether any updates have occurred in the C, D, E ensemble between steps 4 and 5, the re-joining nodes A,B will either receive a TRUNC or a SNAP.

*The problems:*
# If updates have occurred in the C,D,E ensemble, SNAP is sent to the re-joining nodes. This occurs because the code in LearnerHandler.queueCommittedProposals() notices that truncation would cross epochs and bails out, leading to a SNAP being sent. A comment in the code says "We cannot send TRUNC that cross epoch boundary. The learner will crash if it is asked to do so. We will send snapshot this those cases." LearnerHandler.syncFollower() then logs an ERROR saying "Unhandled scenario for peer sid: # fall back to use snapshot" and a comment with this code says "This should never happen, but we should fall back to sending snapshot just in case." Presumably since queueCommittedProposals() is intentionally triggering the snapshot logic, this is not an "unhandled scenario" that warrants logging an ERROR nor is it a case that "should never happen". This inconsistency should be cleaned up. It might also be the case that a TRUNC would work fine in this scenario - see #2 below.
# If no updates have occurred in the C,D,E ensemble, when nodes A,B rejoin LearnerHandler.syncFollower() goes into the "Newer than commitedLog, send trunc and done" clause and sends them a TRUNC. This seems to work fine. However, this would also seem to be a cross-epoch TRUNC, which per the comment discussed above in #1, is expected to cause a crash in the learner. I haven't found anything special about a TRUNC that crosses epochs that would cause a crash in the learner, and I believe that at the time of the TRUNC (or SNAP), the learner is in the same state in both scenarios. It is certainly the case (pending resolution of ZOOKEEPER-1549) that TRUNC is not able to remove data that has been snapshotted, so perhaps detecting “cross-epoch” is a shortcut for trying to detect that scenario? If the resolution of ZOOKEEPER-1549 does not allow TRUNC through a snapshot (or alternately does not allow a benign TRUNC through a snapshot that may not contain uncommitted data), then this case should probably also be a SNAP. If TRUNC is allowed in this case, then perhaps it should also be allowed for case #1, which would be more performant.

*While I certainly could have missed something, it would seem that either both cases should be SNAP or both should be a TRUNC given that the learner is in the same state in both cases*.
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 42 weeks, 3 days ago 0|i2yop3:
ZooKeeper ZOOKEEPER-2435

miss event when the leader stop

Bug Resolved Minor Not A Bug Unassigned BourneHan BourneHan 29/May/16 03:32   31/May/16 21:25 30/May/16 10:56 3.4.6   build   0 2   Hi All,

In my projects, I use three ZooKeeper server as an ensemble:
zk1 as a follower on 192.168.25.221,
zk2 as a follower on 192.168.25.222,
zk3 as the leader on 192.168.25.223.
My two programs using ZooKeepers C client run on 192.168.25.221 and 192.168.25.222.

When watched the ZOO_CONNECTED_STATE, my program will use the zookeeper to obtain a lock do the following:
1. Create a ZOO_EPHEMERAL | ZOO_SEQUENCE node under '/Lock/'.
2. Call getChildren( ) on the '/Lock/' node.
3. If the pathname created in step 1 has the lowest sequence number suffix, the program has the lock and do something,then release the lock simply delete the node created in step 1.
4. The program calls exists() with the watch flag set on the lowest sequence number node.
5. if exists( ) returns false, go to step 2. Otherwise, wait for a notification(ZOO_DELETED_EVENT) for the pathname from the previous step before going to step 2.

When I stop a follower such as zk1/zk2, everything is ok, my programs on 192.168.25.221 and 192.168.25.222 do its work orderly under the lock's control.

When I stop the leader such as zk3(I have restarted zk1/zk2), my program on 192.168.25.221 got the lock and release it normally, and my program on 192.168.25.222 detected existence of the node
created by the program on 192.168.25.221, but keep waiting and can't receive the ZOO_DELETED_EVENT notification.

Does anyone else see the same problem?

PS:
1. The attachment is the log of the zookeeper on 192.168.25.221 and 192.168.25.222 when I stop the leader on 192.168.25.223
2. Actually I have other more programs using ZooKeepers C client run on 192.168.25.221, 192.168.25.222 and 192.168.25.223.
3. The system time on 192.168.25.221 is slower 1 minute and 33 seconds than 192.168.25.222 and 192.168.25.223. so when I stop the leader, it's 2016-05-28 22:33:34 on 192.168.25.221 and 2016-05-28 22:35:07 on 192.168.25.222.
9223372036854775807 No Perforce job exists for this issue. 2 9223372036854775807
3 years, 42 weeks, 1 day ago 0|i2ynvj:
ZooKeeper ZOOKEEPER-2434

Error handling issues in truncation code

Bug Open Minor Unresolved Unassigned Ed Rowe Ed Rowe 28/May/16 21:56   29/May/16 11:39       server   0 5   # If FileTxnLog.truncate() is unable to delete a log file, it calls LOG.warn() but otherwise does nothing. I think this should be a fatal error not a logged warning. Otherwise those log files are going be be read later when the DB is reloaded and data that should have been removed will still be present.
# Learner.syncWithLeader() expects ZKDatabase.truncateLog() to return false on failure, and if this occurs it calls System.exit(). However, this will never happen because ZKDatabase.truncateLog() never returns false - instead an exception is thrown on failure. ZKDatabase.truncateLog() calls FileTxnSnapLog.truncateLog() which calls FileTxnLog.truncate(), each of which is documented to return false on failure but none of which ever does in practice. TruncateTest.testTruncationNullLog() clearly expects an exception on error in ZKDatabase.truncateLog() so there are conflicting expectations in the codebase. It appears that if Learner.syncWithLeader() encounters an exception, System.exit() will _not_ be called and instead we land in the main run loop where we'll start the whole thing again. So there are two things to deal with: a) whether we want to do system.exit or go back to the main run loop if truncation fails, and b) sort out the return false vs. throw exception discrepancy and make it consistent (and change the docs as needed).

I'm happy to propose a patch, but I'd need people with more experience in the codebase to weigh in on the questions above.
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 42 weeks, 4 days ago 0|i2ynnr:
ZooKeeper ZOOKEEPER-2433

ZooKeeperSaslServer: allow user principals in subject

Improvement Closed Major Fixed Andy B Andy B Andy B 23/May/16 05:47   21/Jul/16 16:18 15/Jun/16 16:32 3.5.1 3.5.2, 3.6.0 security   0 8 18000 18000 0% The _createSaslServer_ function in ZooKeeperSaslServer +handles only service principal names+ (eg. *service_name/{color:blue}machine_name{color}@realm*), though sometimes user/service principal names +without host name+ (eg. *service_name@realm*) are used for authentication.

0% 0% 18000 18000 easyfix 9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
Patch
3 years, 40 weeks, 1 day ago
Reviewed
0|i2ycxz:
ZooKeeper ZOOKEEPER-2432

ZOOKEEPER-2431 Document how to use single thread C client API to monitor connection status.

Sub-task Open Major Unresolved Michael Han Michael Han Michael Han 22/May/16 12:43   05/Feb/20 07:17   3.4.8 3.7.0, 3.5.8 c client, documentation   0 1   We need document the correct approach (use watchers combined with zookeeper_interest, select, and zookeeper_process) for monitoring connection events between client and server using single thread C library. See examples https://goo.gl/hql4B1 and https://github.com/fpj/zookeeper-book-example/blob/master/src/main/c/master.c. In particular emphasis that the return code of zookeeper_interest should not be relied for connection monitoring purposes. 9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 43 weeks, 4 days ago 0|i2yc8f:
ZooKeeper ZOOKEEPER-2431

C client documentation improvement

Improvement Open Major Unresolved Michael Han Michael Han Michael Han 22/May/16 12:37   05/Feb/20 07:17   3.4.8 3.7.0, 3.5.8 c client, documentation   1 2   ZOOKEEPER-2432 The existing documentation of C client in Programmer's Guide is incomplete and there are many TBD sections. We should improve the documentation by completing all the TBD sections and add more sample code to demonstrate C client API usage. documentation 9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 43 weeks, 4 days ago 0|i2yc87:
ZooKeeper ZOOKEEPER-2430

Remove jute.maxbuffer limit packetLen in client side

Improvement Open Major Unresolved Yong Zhang Yong Zhang Yong Zhang 21/May/16 04:55   18/Jan/17 18:28           0 2   jute.maxbuffer can be configured in both client and server side, if we try to getChildren
from a parent znode with large number of children znode, client may failed to get because
of packetLen is more than jute.maxbuffer configured.
even if we can change the value in java system property, but we have to restart the application,
and more all data has been in zookeeper client, check the length/size is unnecessary.
9223372036854775807 No Perforce job exists for this issue. 2 9223372036854775807
3 years, 9 weeks, 1 day ago 0|i2ybgv:
ZooKeeper ZOOKEEPER-2429

IbmX509 KeyManager and TrustManager algorithm not supported

Bug Open Minor Unresolved Saurabh jain Saurabh Jain Saurabh Jain 17/May/16 11:19   05/Feb/20 07:16   3.5.0 3.7.0, 3.5.8 security, server   0 3   When connecting from a zookeeper client running in IBM WebSphere Application Server version 8.5.5, with SSL configured in ZooKeeper, the below mentioned exception is observed.

org.jboss.netty.channel.ChannelPipelineException: Failed to initialize a pipeline.
at org.jboss.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:208)
at org.jboss.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:182)
at org.apache.zookeeper.ClientCnxnSocketNetty.connect(ClientCnxnSocketNetty.java:112)
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1130)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1158)
Caused by: org.apache.zookeeper.common.X509Exception$SSLContextException: Failed to create KeyManager
at org.apache.zookeeper.common.X509Util.createSSLContext(X509Util.java:75)
at org.apache.zookeeper.ClientCnxnSocketNetty$ZKClientPipelineFactory.initSSL(ClientCnxnSocketNetty.java:358)
at org.apache.zookeeper.ClientCnxnSocketNetty$ZKClientPipelineFactory.getPipeline(ClientCnxnSocketNetty.java:348)
at org.jboss.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:206)
... 4 more
Caused by: org.apache.zookeeper.common.X509Exception$KeyManagerException: java.security.NoSuchAlgorithmException: SunX509 KeyManagerFactory not available
at org.apache.zookeeper.common.X509Util.createKeyManager(X509Util.java:129)
at org.apache.zookeeper.common.X509Util.createSSLContext(X509Util.java:73)
... 7 more
Caused by: java.security.NoSuchAlgorithmException: SunX509 KeyManagerFactory not available
at sun.security.jca.GetInstance.getInstance(GetInstance.java:172)
at javax.net.ssl.KeyManagerFactory.getInstance(KeyManagerFactory.java:9)
at org.apache.zookeeper.common.X509Util.createKeyManager(X509Util.java:118)


Reason : IBM websphere uses its own jre and supports only IbmX509 keymanager algorithm which is causing an exception when trying to get an key manager instance using SunX509 which is not supported.
Currently KeyManager algorithm name (SunX509) is hardcoded in the class X509Util.java.

Possible fix: Instead of having algorithm name hardcoded to SunX509 we can fall back to the default algorithm supported by the underlying jre.

Instead of having this -
KeyManagerFactory kmf = KeyManagerFactory.getInstance("SunX509");
TrustManagerFactory tmf = TrustManagerFactory.getInstance("SunX509");

can we have ?
KeyManagerFactory kmf = KeyManagerFactory.getInstance(KeyManagerFactory.getDefaultAlgorithm());

TrustManagerFactory tmf = TrustManagerFactory.getInstance(TrustManagerFactory.getDefaultAlgorithm());
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 41 weeks, 5 days ago 0|i2y31r:
ZooKeeper ZOOKEEPER-2422

Improve health reporting by adding heap stats

Improvement Patch Available Major Unresolved Sergey Maslyakov Sergey Maslyakov Sergey Maslyakov 02/May/16 13:41   05/Feb/20 07:11   3.4.8, 3.5.1, 3.6.0 3.7.0, 3.5.8     0 6   In order to improve remote monitoring of the ZooKeeper instance using tools like Icinga/NRPE, it is very desirable to expose JVM heap stats via a light-weight interface. The "mntr" 4lw is a good candidate for this. 9223372036854775807 No Perforce job exists for this issue. 3 9223372036854775807
3 years, 45 weeks, 1 day ago 0|i2x1wf:
ZooKeeper ZOOKEEPER-2421

testSessionReuse is commented out

Bug Open Major Unresolved Prasanth Mathialagan Flavio Paiva Junqueira Flavio Paiva Junqueira 02/May/16 10:56   12/Jun/18 00:43           0 8   This test case in SessionTest:
{noformat}
testSessionReuse
{noformat}
is commented out.
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
1 year, 40 weeks, 2 days ago 0|i2x1lr:
ZooKeeper ZOOKEEPER-2420

Autopurge deletes log file prior to oldest retained snapshot even though restore may need it

Bug Resolved Major Duplicate Ed Rowe Ed Rowe Ed Rowe 02/May/16 01:13   18/Nov/16 02:39 18/Nov/16 02:11     server   0 7   Autopurge retains all log files whose zxid are >= the zxid of the oldest snapshot file that it is going to retain (in PurgeTxnLog retainNRecentSnapshots()). However, unless there is a log file with the same zxid as the oldest snapshot file being retained (and whether log file and snapshot file zxids are equal is timing dependent), loading the database from snapshots/logs will start with the log file _prior_ to the snapshot's zxid. Thus, to avoid data loss autopurge should retain the log file prior to the oldest retained snapshot as well, unless it verifies that it contains no zxids beyond what the snapshot contains or there is a log file whose zxid == snapshot zxid.
9223372036854775807 No Perforce job exists for this issue. 3 9223372036854775807
3 years, 17 weeks, 6 days ago 0|i2x0xb:
ZooKeeper ZOOKEEPER-2419

Zookeeper.log filling up faster due to clients without Auth (KeeperErrorCode = NoAuth)

Improvement Open Blocker Unresolved Unassigned Karthik Shivanna Karthik Shivanna 01/May/16 07:12   16/Jun/19 06:59   3.4.6       0 2   I am seeing that the /var/log/zookeeper/zookeeper.out file is getting filled up faster than usual. It has grown upto 5 GB. When I further saw the out file, lot of them are [INFO] as follows:

2016-03-22 02:03:42,621 - INFO [ProcessThread(sid:4 cport:-1)::PrepRequestProcessor@645] - Got user-level KeeperException when processing sessionid:0x4534413d1f70001 type:create cxid:0x71e0aa99 zxid:0x5f00e3de69 txntype:-1 reqpath:n/a Error Path:null Error:KeeperErrorCode = NoAuth

The log4j properties file was modified to change the parameter for logging from INFO, CONSOLE to INFO, ROLLINGFILE. But I would like to understand where the above INFO is coming from.

Any help is greatly appreciated. Thanks
Zookeeper version: 3.4.6-249--1
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
Important
3 years, 46 weeks ago 0|i2x0j3:
ZooKeeper ZOOKEEPER-2418

txnlog diff sync can skip sending some transactions to followers

Bug Resolved Critical Fixed Brian Nixon Nicholas Wolchko Nicholas Wolchko 28/Apr/16 12:57   03/Jul/19 21:17 03/Jul/19 14:54 3.5.1 3.6.0 server   0 5 604800 598800 6000 0% If the leader is having disk issues so that its on disk txnlog is behind the in memory commit log, it will send a DIFF that is missing the transactions in between the two.

Example:
There are 5 hosts in the cluster. 1 is the leader. 5 is disconnected.
We commit up to zxid 1000.
At zxid 450, the leader's disk stalls, but we still commit transactions because 2,3,4 are up and acking writes.
At zxid 1000, the txnlog on the leader has 1-450 and the commit log has 500-1000.
Then host 5 regains its connection to the cluster and syncs with the leader. It will receive a DIFF containing zxids 1-450 and 500-1000.

This is because queueCommittedProposals in the LearnerHandler just queues everything within its zxid range. It doesn't give an error if there is a gap between peerLastZxid and the iterator it is queueing from.
0% 0% 6000 598800 604800 pull-request-available 9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
37 weeks ago 0|i2wwxr:
ZooKeeper ZOOKEEPER-2416

Remove Java System property usage from ZooKeeper server code

Improvement Open Major Unresolved Mohammad Arshad Mohammad Arshad Mohammad Arshad 26/Apr/16 04:02   05/Feb/20 07:16     3.7.0, 3.5.8 server   0 3   Many ZooKeeper properties are used as Java System properties in the ZooKeeper code.
Some example:
{code}
public static int getSnapCount() {
String sc = System.getProperty("zookeeper.snapCount");
{code}
{code}
public int getGlobalOutstandingLimit() { String sc =
System.getProperty("zookeeper.globalOutstandingLimit");
{code}
Using ZooKeeper properties as Java system properties causes following
problems
# Can not create two or more ZooKeeper Server in a single JVM
with different properties for testing
# The properties initialization and validation is very much mixed with actual business logic which should not be the case.
ZOOKEEPER-2139 removed the ZooKeeper client side Java System properties so as part of this jira handling only ZooKeeper server side Java System properties to be removed.
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 47 weeks, 2 days ago 0|i2wqjj:
ZooKeeper ZOOKEEPER-2415

SessionTest is using Thread deprecated API.

Test Closed Major Fixed Andor Molnar Flavio Paiva Junqueira Flavio Paiva Junqueira 24/Apr/16 10:30   04/Oct/19 10:55 24/Apr/18 19:49 3.4.8, 3.5.1, 3.6.0 3.5.4, 3.6.0, 3.4.13 tests   0 4   The test class is using calls such as {{Thread.suspend}} and {{Thread.resume}}. 9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
1 year, 47 weeks, 1 day ago
Reviewed
0|i2wnxz:
ZooKeeper ZOOKEEPER-2414

c-client aborted when operate's path is invalid in zoo_amulti

Bug Patch Available Major Unresolved Meyer Kizner Tianyi Zhang Tianyi Zhang 21/Apr/16 00:07   14/Dec/19 06:08   3.4.6, 3.4.8 3.7.0 c client   0 5   code like this:
{code}
zoo_op_t ops[2];
zoo_op_result_t results[2];
zoo_create_op_init(ops, "test", "1", 1, &ZOO_OPEN_ACL_UNSAFE, 0, NULL, 0);
zoo_create_op_init(ops+1, "/test/1", "2", 1, &ZOO_OPEN_ACL_UNSAFE, 0, NULL, 0);
zoo_multi(zkhandle, 2, ops, results);
{code}
The ops->path is invalid, and it will cause double free in the line 3136 of zookeeper.c.
{code}
for (index=0; index < count; index++) {
const zoo_op_t *op = ops+index;
zoo_op_result_t *result = results+index;
completion_list_t *entry = NULL;

struct MultiHeader mh = { STRUCT_INITIALIZER(type, op->type), STRUCT_INITIALIZER(done, 0), STRUCT_INITIALIZER(err, -1) };
rc = rc < 0 ? rc : serialize_MultiHeader(oa, "multiheader", &mh);

switch(op->type) {
case ZOO_CREATE_OP: {
struct CreateRequest req;

rc = rc < 0 ? rc : CreateRequest_init(zh, &req,
op->create_op.path, op->create_op.data,
op->create_op.datalen, op->create_op.acl,
op->create_op.flags);
rc = rc < 0 ? rc : serialize_CreateRequest(oa, "req", &req);
result->value = op->create_op.buf;
result->valuelen = op->create_op.buflen;

enter_critical(zh);
entry = create_completion_entry(h.xid, COMPLETION_STRING, op_result_string_completion, result, 0, 0);
leave_critical(zh);
--> free_duplicate_path(req.path, op->create_op.path);
break;
}
{code}
This problem will happen when the 'rc' of last op is less than 0(maybe ZBADARGUMENTS or ZINVALIDSTATE).
In my case, rc of op[0] is ZBADARGUMENTS, and the req.path of the ‘free_duplicate_path’ is still 'test' when execute op[1].
I‘m confused about why not break the for-loop when the 'rc' is less than 0?
9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
1 year, 20 weeks, 6 days ago 0|i2wf6f:
ZooKeeper ZOOKEEPER-2413

ContainerManager doesn't close the Timer it creates when stop() is called

Bug Closed Major Fixed Jordan Zimmerman Jordan Zimmerman Jordan Zimmerman 13/Apr/16 14:35   21/Jul/16 16:18 24/Apr/16 17:22 3.5.1 3.5.2, 3.6.0 server   0 5   ContainerManager creates a Timer object. It's stop() method cancel's the running task but doesn't close the Timer itself. This ends up leaking a Thread (internal to the Timer). 9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
3 years, 47 weeks, 4 days ago Thanks, Jordan.

Branch 3.5: Committed revision 1740737.
Trunk: Committed revision 1740738.
0|i2w2nz:
ZooKeeper ZOOKEEPER-2412

leader zk out of memory, and leader db lastZxid is not update when process set data.

Bug Resolved Major Duplicate Unassigned Yongcheng Liu Yongcheng Liu 12/Apr/16 09:32   10/May/16 09:57 10/May/16 09:57 3.4.6 3.4.8     0 4   3 zk, the follower (myid=6) server is injected into the network delay fault for test, so this follower(id=6) data is behind leader, leader need send SNAP to this follower every time. the time begin to have problem is 20:59:47

(1) out of memory log:

2016-03-24 23:01:21,355 [myid:4] - INFO [LearnerHandler-/192.168.50.26:35112:LearnerHandler@330] - Follower sid: 6 : info : org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@5cf112b0
2016-03-24 23:01:21,355 [myid:4] - INFO [LearnerHandler-/192.168.50.26:35112:LearnerHandler@385] - Synchronizing with Follower sid: 6 maxCommittedLog=0x9000033a1 minCommittedLog=0x9000031ad peerLastZxid=0x90000280a
2016-03-24 23:01:21,355 [myid:4] - WARN [LearnerHandler-/192.168.50.26:35112:LearnerHandler@446] - Unhandled proposal scenario
2016-03-24 23:01:21,355 [myid:4] - INFO [LearnerHandler-/192.168.50.26:35112:LearnerHandler@462] - Sending SNAP
2016-03-24 23:01:21,893 [myid:4] - INFO [NIOServerCxn.Factory:/192.168.50.24:10540:NIOServerCnxnFactory@207] - Current connection (from /192.168.50.22 Cnxns = 4; totalCnxns = 15)
2016-03-24 23:01:22,625 [myid:4] - WARN [NIOServerCxn.Factory:/192.168.50.24:10540:ZooKeeperServer@832] - Connection request from old client /192.168.50.22:49695; will be dropped if server is in r-o mode
2016-03-24 23:01:23,283 [myid:4] - INFO [QuorumPeer[myid=4]/192.168.50.24:10540:Leader@493] - Shutting down
2016-03-24 23:01:24,102 [myid:4] - INFO [QuorumPeer[myid=4]/192.168.50.24:10540:Leader@499] - Shutdown called
2016-03-24 23:01:24,040 [myid:4] - INFO [SessionTracker:ZooKeeperServer@347] - Expiring session 0x453a6dc5b7a007e, timeout of 3500ms exceeded

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "NIOServerCxn.Factory:/192.168.50.24:10540"
2016-03-24 23:01:25,001 [myid:4] - WARN [QuorumPeer[myid=4]/192.168.50.24:10540:QuorumPeer@827] - QuorumPeer main thread exited

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "LearnerHandler-/192.168.50.26:35112"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "QuorumPeer[myid=4]/192.168.50.24:10540"
2016-03-24 23:01:24,227 [myid:4] - ERROR [LearnerHandler-/192.168.50.26:35112:NIOServerCnxnFactory$1@44] - Thread LearnerHandler Socket[addr=/192.168.50.26,port=35112,localport=10550] tickOfNextAckDeadline:38310 synced?:true queuedPacketLength:7910 died
2016-03-24 23:01:28,492 [myid:4] - INFO [main:QuorumPeerMain@93] - Exiting normally



(2) this leader is very strange, db lastzxid not update, log as follow (after grep "Synchronizing with"), we can see max commit zxid from leader is not update any more. From the beginning of 20:59:47, leader lastZxid not update.

2016-03-24 20:56:10,266 [myid:4] - INFO [LearnerHandler-/192.168.50.26:48439:LearnerHandler@385] - Synchronizing with Follower sid: 6 maxCommittedLog=0x90000333f minCommittedLog=0x90000314b peerLastZxid=0x90000280a
2016-03-24 20:57:59,203 [myid:4] - INFO [LearnerHandler-/192.168.50.26:46956:LearnerHandler@385] - Synchronizing with Follower sid: 6 maxCommittedLog=0x900003398 minCommittedLog=0x9000031a4 peerLastZxid=0x90000280a
2016-03-24 20:59:47,928 [myid:4] - INFO [LearnerHandler-/192.168.50.26:20601:LearnerHandler@385] - Synchronizing with Follower sid: 6 maxCommittedLog=0x9000033a1 minCommittedLog=0x9000031ad peerLastZxid=0x90000280a
2016-03-24 21:01:26,475 [myid:4] - INFO [LearnerHandler-/192.168.50.26:29622:LearnerHandler@385] - Synchronizing with Follower sid: 6 maxCommittedLog=0x9000033a1 minCommittedLog=0x9000031ad peerLastZxid=0x90000280a
2016-03-24 21:03:16,552 [myid:4] - INFO [LearnerHandler-/192.168.50.26:35717:LearnerHandler@385] - Synchronizing with Follower sid: 6 maxCommittedLog=0x9000033a1 minCommittedLog=0x9000031ad peerLastZxid=0x90000280a
2016-03-24 21:03:44,427 [myid:4] - INFO [LearnerHandler-/192.168.50.26:48197:LearnerHandler@385] - Synchronizing with Follower sid: 6 maxCommittedLog=0x9000033a1 minCommittedLog=0x9000031ad peerLastZxid=0x90000280a
2016-03-24 21:05:01,125 [myid:4] - INFO [LearnerHandler-/192.168.50.26:57826:LearnerHandler@385] - Synchronizing with Follower sid: 6 maxCommittedLog=0x9000033a1 minCommittedLog=0x9000031ad peerLastZxid=0x90000280a
2016-03-24 21:06:54,187 [myid:4] - INFO [LearnerHandler-/192.168.50.26:30137:LearnerHandler@385] - Synchronizing with Follower sid: 6 maxCommittedLog=0x9000033a1 minCommittedLog=0x9000031ad peerLastZxid=0x90000280a
2016-03-24 21:07:42,780 [myid:4] - INFO [LearnerHandler-/192.168.50.26:24255:LearnerHandler@385] - Synchronizing with Follower sid: 6 maxCommittedLog=0x9000033a1 minCommittedLog=0x9000031ad peerLastZxid=0x90000280a
2016-03-24 21:08:41,279 [myid:4] - INFO [LearnerHandler-/192.168.50.26:40909:LearnerHandler@385] - Synchronizing with Follower sid: 6 maxCommittedLog=0x9000033a1 minCommittedLog=0x9000031ad peerLastZxid=0x90000280a
2016-03-24 21:10:23,137 [myid:4] - INFO [LearnerHandler-/192.168.50.26:64166:LearnerHandler@385] - Synchronizing with Follower sid: 6 maxCommittedLog=0x9000033a1 minCommittedLog=0x9000031ad peerLastZxid=0x90000280a
2016-03-24 21:11:50,003 [myid:4] - INFO [LearnerHandler-/192.168.50.26:56070:LearnerHandler@385] - Synchronizing with Follower sid: 6 maxCommittedLog=0x9000033a1 minCommittedLog=0x9000031ad peerLastZxid=0x90000280a
2016-03-24 21:12:11,956 [myid:4] - INFO [LearnerHandler-/192.168.50.26:41423:LearnerHandler@385] - Synchronizing with Follower sid: 6 maxCommittedLog=0x9000033a1 minCommittedLog=0x9000031ad peerLastZxid=0x90000280a
2016-03-24 21:13:08,286 [myid:4] - INFO [LearnerHandler-/192.168.50.26:26757:LearnerHandler@385] - Synchronizing with Follower sid: 6 maxCommittedLog=0x9000033a1 minCommittedLog=0x9000031ad peerLastZxid=0x90000280a
2016-03-24 21:13:59,960 [myid:4] - INFO [LearnerHandler-/192.168.50.26:62785:LearnerHandler@385] - Synchronizing with Follower sid: 6 maxCommittedLog=0x9000033a1 minCommittedLog=0x9000031ad peerLastZxid=0x90000280a
2016-03-24 21:15:41,103 [myid:4] - INFO [LearnerHandler-/192.168.50.26:53141:LearnerHandler@385] - Synchronizing with Follower sid: 6 maxCommittedLog=0x9000033a1 minCommittedLog=0x9000031ad peerLastZxid=0x90000280a
2016-03-24 21:16:11,125 [myid:4] - INFO [LearnerHandler-/192.168.50.26:39551:LearnerHandler@385] - Synchronizing with Follower sid: 6 maxCommittedLog=0x9000033a1 minCommittedLog=0x9000031ad peerLastZxid=0x90000280a
2016-03-24 21:17:25,541 [myid:4] - INFO [LearnerHandler-/192.168.50.26:24638:LearnerHandler@385] - Synchronizing with Follower sid: 6 maxCommittedLog=0x9000033a1 minCommittedLog=0x9000031ad peerLastZxid=0x90000280a
2016-03-24 21:18:25,039 [myid:4] - INFO [LearnerHandler-/192.168.50.26:54723:LearnerHandler@385] - Synchronizing with Follower sid: 6 maxCommittedLog=0x9000033a1 minCommittedLog=0x9000031ad peerLastZxid=0x90000280a
2016-03-24 21:19:04,148 [myid:4] - INFO [LearnerHandler-/192.168.50.26:37450:LearnerHandler@385] - Synchronizing with Follower sid: 6 maxCommittedLog=0x9000033a1 minCommittedLog=0x9000031ad peerLastZxid=0x90000280a


(3) we can see leader Snapshotting to the same file in different time, but receive different TxnZxid(0x900003b70 and 0x9000049dd), this show leader has not been updated lastZxid in db.

Snapshotting 1:
2016-03-24 21:33:27,214 [myid:4] - INFO [Snapshot Thread:FileTxnSnapLog@253] - Snapshotting: 0x9000033a1 to /opt/dsware/agent/zk/data/version-2/snapshot.9000033a1
2016-03-24 21:33:27,333 [myid:4] - INFO [SyncThread:4:FileTxnLog@199] - Creating new log file: log.900003b70

Snapshotting 2:
2016-03-24 22:41:26,601 [myid:4] - INFO [Snapshot Thread:FileTxnSnapLog@253] - Snapshotting: 0x9000033a1 to /opt/dsware/agent/zk/data/version-2/snapshot.9000033a1
2016-03-24 22:41:26,662 [myid:4] - INFO [SyncThread:4:FileTxnLog@199] - Creating new log file: log.9000049dd


(4) finally, this node(leader server) zxid is behind zk c client, log as follow:

2016-03-24 23:00:53,712 [myid:4] - WARN [NIOServerCxn.Factory:/192.168.50.24:10540:ZooKeeperServer@832] - Connection request from old client /192.168.50.23:35043; will be dropped if server is in r-o mode
2016-03-24 23:00:53,713 [myid:4] - INFO [NIOServerCxn.Factory:/192.168.50.24:10540:ZooKeeperServer@851] - Refusing session request for client /192.168.50.23:35043 as it has seen zxid 0x900004e1f our last zxid is 0x9000033a1 client must try another server
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 45 weeks, 2 days ago the same as https://issues.apache.org/jira/browse/ZOOKEEPER-2201 0|i2vzxz:
ZooKeeper ZOOKEEPER-2411

email

Bug Resolved Major Invalid Unassigned André Cruz André Cruz 11/Apr/16 05:03   11/Apr/16 05:04 11/Apr/16 05:04         0 1   9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 49 weeks, 3 days ago 0|i2vx0f:
ZooKeeper ZOOKEEPER-2410

add time unit to 'ELECTION TOOK' log.info message

Improvement Closed Trivial Fixed Christine Poerschke Christine Poerschke Christine Poerschke 01/Apr/16 14:20   21/Jul/16 16:18 01/Jun/16 19:01   3.5.2, 3.6.0 leaderElection, quorum, server   0 4   A three-line change to add the time unit to 'ELECTION TOOK' log.info message to help people not yet so familiar with zookeeper interpret log files. 9223372036854775807 No Perforce job exists for this issue. 2 9223372036854775807
Patch
3 years, 42 weeks, 1 day ago
Reviewed
0|i2vj3j:
ZooKeeper ZOOKEEPER-2409

zookeeper recipes lock's c implement: function child_floor bug

Bug Open Major Unresolved Unassigned europelee europelee 01/Apr/16 02:59   01/Apr/16 02:59   3.4.8   c client   1 2 3600 3600 0% zookeeper cluster with multi servers(not standalone mode), and multi clients connect to different zookeeper server. 1. start zookeeper cluster with multi servers
2. multi clients connect to different zookeeper server at the same time
3. clients use lock from zookeeper recipes lock's C implement.

for example:
Client A create a node like x-025373e3a9960050-0000000067
and Client B create a node like x-015373e3a9960050-0000000068;
A is a lock owner now, then kill A, as expect, B should become owner, but in fact B not.
Because in zoo_lock.c, function zkr_lock_operation call child_floor to monitoring a pre node, but child_floor has bug, it caused B not check its prenode A.

B function child_floor just simply strcmp "x-025373e3a9960050-0000000067" with own node "x-015373e3a9960050-0000000068",
it should only strcmp "0000000067" with "0000000068", not include session info.
besides, it is better that using binary search than travelling every node for looking for a pre node when there exists many nodes.
fix:
static char* child_floor(char **sorted_data, int len, char *element) {
char* ret = NULL;

int begin = 0;
int end = len-1;
int index = 0;

while (begin <= end) {
index = (begin+end)/2;
int iCmpRet = strcmp(strrchr(sorted_data[index], '-')+1, strrchr(element, '-')+1);
if (iCmpRet < 0) {
begin = index + 1;
}
else {
if (iCmpRet == 0) {
if (index - 1 >= 0) {
ret = sorted_data[index-1];
}
break;
}
else {
end = index - 1;
}
}
}

return ret;
}
0% 0% 3600 3600 easyfix 9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 50 weeks, 6 days ago 0|i2vi27:
ZooKeeper ZOOKEEPER-2408

zoo_awget() ctx memory does not be freed if callback watcher was not invoked

Bug Open Major Unresolved Unassigned Kevin Kevin 31/Mar/16 22:44   31/Mar/16 22:44   3.4.6   c client   0 1   centos 7 I lunch a client, then watch some node, call zoo_awget() to watch data, when the client exit, if the node data doesn't change, the callback 'watcher' won't be invoked. and the memory of watcherCtx is not freed

ZOOAPI int zoo_awget(zhandle_t *zh, const char *path,
watcher_fn watcher, void* watcherCtx,
data_completion_t completion, const void *data);

I use valgrind to check and the result show the memory lost.
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 50 weeks, 6 days ago 0|i2vhtj:
ZooKeeper ZOOKEEPER-2407

EventThread in ClientCnxn can't be closed when SendThread exits because of auth failed during reconnection

Bug Open Major Unresolved sunhaitao sunhaitao sunhaitao 30/Mar/16 07:21   05/Feb/20 07:16   3.5.1 3.7.0, 3.5.8     0 4   EventThread in ClientCnxn can't be closed when SendThread exits because of auth failed during reconnection.
for send thread if it is in authfailed state, the send thread exits,but the event thread is still running.

observation:
use jstack tho check the thread running they find the send thread no longer exists but event thread is still threre
even when we call zookeeper.close(),the eventthread is still there.

Stack trace:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:514)
9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
1 year, 17 weeks, 1 day ago 0|i2ve8n:
ZooKeeper ZOOKEEPER-2406

/etc/zookeeper isn't used for ZOOCFGDIR

Bug Open Minor Unresolved Unassigned Mark Elrod Mark Elrod 29/Mar/16 15:35   29/Mar/16 15:35   3.5.1       0 2   The comment in zkEnv.sh indicates that /etc/zookeeper should be an option for the ZOOCFGDIR but the code beneath it does not look to see if it exists:

{noformat}
# We use ZOOCFGDIR if defined,
# otherwise we use /etc/zookeeper
# or the conf directory that is
# a sibling of this script's directory


ZOOBINDIR="${ZOOBINDIR:-/usr/bin}"
ZOOKEEPER_PREFIX="${ZOOBINDIR}/.."

if [ "x$ZOOCFGDIR" = "x" ]
then
if [ -e "${ZOOKEEPER_PREFIX}/conf" ]; then
ZOOCFGDIR="$ZOOBINDIR/../conf"
else
ZOOCFGDIR="$ZOOBINDIR/../etc/zookeeper"
fi
fi
{noformat}

Should this be something like:

{noformat}
if [ "x$ZOOCFGDIR" = "x" ]
then
if [ -e "/etc/zookeeper" ]; then
ZOOCFGDIR="/etc/zookeeper"
elif [ -e "${ZOOKEEPER_PREFIX}/conf" ]; then
ZOOCFGDIR="$ZOOBINDIR/../conf"
else
ZOOCFGDIR="$ZOOBINDIR/../etc/zookeeper"
fi
fi
{noformat}

I am not sure if ZOOBINDIR/../etc/zookeeper is supposed to be an option or a typo but in the default setup ZOOBINDIR/../conf exists so even if it were changed to /etc/zookeeper it would never try to use it.
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 51 weeks, 2 days ago 0|i2vd2n:
ZooKeeper ZOOKEEPER-2405

getTGT() in Login.java mishandles confidential information

Bug Closed Blocker Fixed Michael Han Patrick D. Hunt Patrick D. Hunt 26/Mar/16 15:04   21/Jul/16 16:18 25/May/16 16:47 3.4.8, 3.5.1, 3.6.0 3.4.9, 3.5.2, 3.6.0 kerberos, security, server   0 5   We're logging the kerberos ticket when in debug mode, probably not the best idea. This was identified as a "critical" issue by Fortify.

{noformat}
for(KerberosTicket ticket: tickets) {
KerberosPrincipal server = ticket.getServer();
if (server.getName().equals("krbtgt/" + server.getRealm() + "@" + server.getRealm())) {
LOG.debug("Found tgt " + ticket + ".");
return ticket;
}
}
{noformat}
9223372036854775807 No Perforce job exists for this issue. 5 9223372036854775807
3 years, 43 weeks, 1 day ago
Reviewed
0|i2v96v:
ZooKeeper ZOOKEEPER-2404

Sections to be completed in Programmer's guide document.

Task Open Trivial Unresolved Unassigned Ibrahim Ibrahim 24/Mar/16 11:11   24/Mar/16 11:19   4.0.0   documentation   1 2 7257600 7257600 0% There are some sections in Programmer's guide document need to be completed, such as Read Operations, Write operations and Connecting to Zookeeper.
0% 0% 7257600 7257600 documentation 9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
4 years ago 0|i2v5vb:
ZooKeeper ZOOKEEPER-2403

Consistently handle and document boolean properties (true/false as well as yes/no)

Wish Open Trivial Unresolved Ryan P Ryan P Ryan P 23/Mar/16 16:21   11/May/16 11:44           0 3   Currently zookeeper.skipACL is evaluated to be either yes or no. This is less than intuitive most developers would expect this to except true or false.

https://github.com/apache/zookeeper/blob/trunk/src/java/main/org/apache/zookeeper/server/PrepRequestProcessor.java#L96-Lundefined


9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
3 years, 45 weeks, 1 day ago 0|i2v473:
ZooKeeper ZOOKEEPER-2402

Document client side properties

Improvement Closed Major Fixed Mohammad Arshad Mohammad Arshad Mohammad Arshad 23/Mar/16 08:09   29/Sep/16 17:42 23/May/16 17:34   3.5.2, 3.6.0 documentation   0 5   There are many ZooKeeper java client properties which are not documented.
Bellow client properties are not documented in Admin Guide
# zookeeper.sasl.client.username
# zookeeper.sasl.clientconfig
# zookeeper.sasl.client
# zookeeper.server.realm
# zookeeper.disableAutoWatchReset

Bellow two client properties are documented in Admin Guide but these are documented in server configuration section
# zookeeper.client.secure
# zookeeper.clientCnxnSocket
9223372036854775807 No Perforce job exists for this issue. 4 9223372036854775807
3 years, 43 weeks, 3 days ago 0|i2v35z:
ZooKeeper ZOOKEEPER-2401

Document leader.nodelay and follower.nodelay ZooKeeper server properties

Bug Open Minor Unresolved Mohammad Arshad Mohammad Arshad Mohammad Arshad 23/Mar/16 07:40   23/Mar/16 07:40       documentation   0 1   Following properties used in the zookeeper server, also configurable, but not documented in admin guide.
# leader.nodelay
# follower.nodelay
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
4 years, 1 day ago 0|i2v353:
ZooKeeper ZOOKEEPER-2400

ZooKeeper not starting: Follower is ahead of the leader

Bug Resolved Major Not A Problem Unassigned Andrey Andrey 23/Mar/16 05:00   23/Mar/16 17:31 23/Mar/16 17:31 3.4.6   quorum   0 2   Steps to reproduce:
# Select deprecated algorithm in zoo.cfg: {code}electionAlg=0{code}
# Start zookeeper cluster: A(index 3),B(index 1),C(index 2) nodes
# Stop A node.
# Make some change to zk data. i.e. re-create ephemeral node. Make sure currentEpoch increased in B and C nodes.
# currentEpoch/acceptedEpoch in node A less than B/C epoch
# Stop node B. Zookeeper cluster is not available
# Start node A. In A's node logs:
{code}
LEADING [quorum.QuorumPeer] [QuorumPeer[myid=3]/0:0:0:0:0:0:0:0:15523]
LEADING - LEADER ELECTION TOOK - 1458721180995 [quorum.Leader]
Follower sid: 2 : info : org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@3a888c1
java.io.IOException: Follower is ahead of the leader, leader summary: 10 (current epoch), 42949672964 (last zxid)
at org.apache.zookeeper.server.quorum.Leader.waitForEpochAck(Leader.java:894)
at org.apache.zookeeper.server.quorum.LearnerHandler.run(LearnerHandler.java:365)

...

Follower sid: 1 : info : org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@5001b9f5

...

java.lang.InterruptedException: Timeout while waiting for epoch to be acked by quorum
at org.apache.zookeeper.server.quorum.Leader.waitForEpochAck(Leader.java:915)
at org.apache.zookeeper.server.quorum.Leader.lead(Leader.java:394)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:799)
{code}

The logs above will be printed indefinitely and cluster won't recover.
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
4 years, 1 day ago 0|i2v2uf:
ZooKeeper ZOOKEEPER-2399

Partial Initial Patches to update Zookeeper trunk to Netty 4.x API

Improvement Resolved Major Fixed Unassigned William L Thomson Jr William L Thomson Jr 22/Mar/16 18:51   22/Jul/19 00:27 16/Jul/19 16:38         1 5   These are initial patches to update Zookeeper to Netty 4.x API. They are not complete, as I am not familiar with Zookeeper or Netty. I just used Netty documents that detailed the difference in the API to make the changes. I believe I am 80-90% of the way there, but need someone familiar with Zookeeper and/or Netty to finish the last part and make sure it actually works and I did not fubar Zookeeper :) 9223372036854775807 No Perforce job exists for this issue. 3 9223372036854775807
34 weeks, 3 days ago 0|i2v22n:
ZooKeeper ZOOKEEPER-2398

Config options should not be only available via system property

Improvement Open Minor Unresolved Unassigned Jason Rosenberg Jason Rosenberg 22/Mar/16 09:11   05/Feb/20 07:16     3.7.0, 3.5.8     0 2   Some config options (such as enabling readonly mode) are only settable via a system property. This feels clunky, and makes it less seamless for testing, or for apps which embed a ZooKeeper inside a java container, etc.

I ran into this issue specifically in the case of creating unit tests to test read-only mode client side behavior. In this case, I want to run multiple QuorumPeer's in the same jvm, and have some of them enabled for read-only and some not enabled. This is not possible with the current System.setProperty approach.

In general, I question the need for using system properties for configuration, since it makes embedding a server within a dependency injection framework more difficult, and is in general less easy to integrate into generic deployment systems.
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
4 years, 2 days ago 0|i2v0vz:
ZooKeeper ZOOKEEPER-2397

ZOOKEEPER-2314 Undocumented SASL properties

Sub-task Open Major Unresolved Flavio Paiva Junqueira Flavio Paiva Junqueira Flavio Paiva Junqueira 21/Mar/16 18:59   22/Jun/18 00:49   3.4.8, 3.5.1, 3.4.11   documentation   0 5   There are a number of properties spread across the code that do not appear in the docs. For example, zookeeper.allowSaslFailedClients isn't documented afaict. 9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
3 years, 24 weeks, 3 days ago 0|i2uzq7:
ZooKeeper ZOOKEEPER-2396

ZOOKEEPER-2314 Login object in ZooKeeperSaslClient is static

Sub-task Closed Major Fixed Flavio Paiva Junqueira Flavio Paiva Junqueira Flavio Paiva Junqueira 21/Mar/16 18:58   21/Jul/16 16:18 07/May/16 11:51   3.4.9, 3.5.2, 3.6.0 documentation   0 4   The login object in ZooKeeperSaslClient is static, which means that if you try to create another client for tests, the login object will be the first one you've set for all runs. I've experienced this with 3.4.6. 9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 45 weeks, 5 days ago 0|i2uzpz:
ZooKeeper ZOOKEEPER-2395

allow ant command line control of junit test jvm args

Improvement Open Major Unresolved Patrick D. Hunt Patrick D. Hunt Patrick D. Hunt 21/Mar/16 18:48   05/Feb/20 07:16     3.7.0, 3.5.8 build, tests   0 2   We're seeing some failing jobs (see below) and the speculation is that it might be due to ipv6 vs ipv4 usage. It would be nice to turn on "prefer ipv4" in the jvm but there is no easy way to do that. I'll propose that we add a variable to ant that's passed through to the jvm.

----
This is very odd. It failed 2 of the last three times it was run on H9
with the following:

2016-03-20 06:06:18,480 [myid:] - INFO
[main:JUnit4ZKTestRunner$LoggedInvokeMethod@74] - TEST METHOD FAILED
testBindByAddress
java.net.SocketException: No such device
at java.net.NetworkInterface.isLoopback0(Native Method)
at java.net.NetworkInterface.isLoopback(NetworkInterface.java:339)
at org.apache.zookeeper.test.ClientPortBindTest.testBindByAddress(ClientPortBindTest.java:61)

https://builds.apache.org/job/ZooKeeper_branch34/buildTimeTrend

Why would it pass one of the times though if there is no loopback
device on the host? That seems very odd!
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 8 weeks ago 0|i2uzov:
ZooKeeper ZOOKEEPER-2394

Many ZooKeeper properties are not configurable in zoo.cfg

Improvement Patch Available Major Unresolved Mohammad Arshad Mohammad Arshad Mohammad Arshad 21/Mar/16 05:38   05/Feb/20 07:11     3.7.0, 3.5.8 server   0 2   Many ZooKeeper properties are not configurable in zoo.cfg. If configured in zoo.cfg, {{QuorumPeerConfig}} parse logic will pre append "zookeeper" which is not the same property used in code. So a property with a name {{abc.xyz}} becomes {{zookeeper.abc.xyz}}
Bellow are properties which can not configured in zoo.cfg, can only be configured using java system properties.

# follower.nodelay
# leader.nodelay
# readonlymode.enabled
# jute.maxbuffer
# znode.container.checkIntervalMs
# znode.container.maxPerMinute

This jira targets to make these properties configurable in zoo.cfg as well.
9223372036854775807 No Perforce job exists for this issue. 2 9223372036854775807
3 years, 39 weeks, 2 days ago 0|i2uy3z:
ZooKeeper ZOOKEEPER-2393

Revert run-time dependency on log4j and slf4j-log4j12

Bug Closed Blocker Fixed Mohammad Arshad Mohammad Arshad Mohammad Arshad 17/Mar/16 21:39   21/Jul/16 16:18 19/Mar/16 17:56   3.5.2, 3.6.0 server   0 4   Zookeeper run-time dependency on log4j and slf4j-log4j12 was removed as part of ZOOKEEPER-1371 jira work.
Following things were done as part of ZOOKEEPER-1371
# Removed direct log4j API use from the code, instead used slf4j-api
# Changed log4j and slf4j-log4j12 run time dependency to test time dependency
# Upgraded log4j, slf4j-log4j12 and slf4j-api versions.
Here is the component wise version change
#* (zookeeper)ivy.xml
log4j: 1.2.15 -->1.7.5
#* src\contrib\loggraph\ivy.xml
slf4j-api: 1.6.1 -->1.7.5
slf4j-log4j12: 1.6.1 -->1.7.5
log4j: 1.2.15 -->1.7.5
#* src\contrib\rest\ivy.xml
slf4j-api: 1.6.1 -->1.7.5
slf4j-log4j12: 1.6.1 -->1.7.5
log4j: 1.2.15 -->1.7.5
#* src\contrib\zooinspector\ivy.xml
slf4j-api: 1.6.1 -->1.7.5
slf4j-log4j12: 1.6.1 -->1.7.5
log4j: 1.2.15 -->1.7.5

The major problem with ZOOKEEPER-1371 change is that it removed run time dependency. For more detail refer ZOOKEEPER-2342 discussion
Now as part of this jira revert back only run time dependency, #2, on log4j and slf4j-log4j12.


9223372036854775807 No Perforce job exists for this issue. 2 9223372036854775807
4 years, 5 days ago
Reviewed
0|i2uv3b:
ZooKeeper ZOOKEEPER-2392

Update netty to 3.7.1.Final

Improvement Closed Minor Fixed Hendy Irawan Hendy Irawan Hendy Irawan 16/Mar/16 10:50   21/Jul/16 16:18 18/Mar/16 13:25 3.4.6, 3.5.1 3.5.2, 3.6.0 build   0 2   9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
Patch
4 years, 6 days ago
Reviewed
https://github.com/apache/zookeeper/pull/57 0|i2urof:
ZooKeeper ZOOKEEPER-2391

setMin/MaxSessionTimeout of ZookeeperServer are implemented in quite a weak way

Bug Patch Available Minor Unresolved Kazuaki Banzai Kazuaki Banzai Kazuaki Banzai 16/Mar/16 08:34   12/Dec/16 03:39       server   0 3   setMin/MaxSessionTimeout of ZookeeperServer are implemented in quite a weak way.
* -1 restores the default, but this is not documented.
* values < -1 are permitted but make no sense.
* min > max is permitted but makes not sense.
9223372036854775807 No Perforce job exists for this issue. 6 9223372036854775807
3 years, 14 weeks, 3 days ago 0|i2urhz:
ZooKeeper ZOOKEEPER-2390

pom file needs license information

Improvement Resolved Blocker Duplicate Patrick D. Hunt Ben McCann Ben McCann 15/Mar/16 21:46   17/Mar/16 02:30 17/Mar/16 02:30 3.5.1, 3.5.2, 3.6.0, 4.0.0   build   0 1   Can you add a license section to the pom file?

org.apache.zookeeper:zookeeper:3.5.1-alpha does specify one currently

Automated tools utilize the license in the pom the ensure we're clients are using appropriately licensed software

<licenses>
<license>
<name>Apache License, Version 2.0</name>
<url>http://www.apache.org/licenses/LICENSE-2.0.txt</url>
<distribution>repo</distribution>
</license>
</licenses>
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
4 years, 1 week ago 0|i2uqsf:
ZooKeeper ZOOKEEPER-2389

read-only observer doesn't load transaction log when transitioning to read-only

Bug Open Major Unresolved Unassigned Jason Rosenberg Jason Rosenberg 14/Mar/16 20:02   22/Jun/18 00:49   3.4.8, 3.4.11       0 1   I have rediscovered an issue, that was apparently posted a while back (link below). It seems that if I configure an Observer node to be enabled for read-only mode, with syncEnabled = true, it properly syncs its transaction log with the quorum. However, if I shut down the quorum participants, and the Observer automatically transitions to read-only mode, it does not load the saved transaction log, and thus rejects any client connection with a zxid > 0. But If I restart the Observer node, it reloads it's persisted transaction log and serves read-only requests at the latest zxid. Is this the correct behavior? Things run fine if instead of an observer, I do the same with a read-only participant. In this case, it transitions without issue to a read-only server, and serves the current transaction log.

It seems to me this issue renders read-only observers completely useless. What am I missing here?

I'm seeing this with 3.4.8

It seems this was discovered and reported a long time ago here:
http://grokbase.com/t/zookeeper/user/14c16b1d22/issue-with-zxid-during-observer-failover-to-read-only
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
4 years, 1 week, 3 days ago 0|i2un47:
ZooKeeper ZOOKEEPER-2388

Unit tests failing on Solaris

Bug Closed Blocker Fixed Mohammad Arshad Patrick D. Hunt Patrick D. Hunt 14/Mar/16 17:06   21/Jul/16 16:18 17/Mar/16 02:16 3.5.2 3.5.2, 3.6.0 tests   0 4   The same two tests are failing consistently on Solaris in 3.5/trunk (I don't see similar failures in 3.4, jenkins is mostly green there)

org.apache.zookeeper.server.quorum.LocalPeerBeanTest.testClientAddress
org.apache.zookeeper.server.quorum.QuorumPeerTest.testQuorumPeerListendOnSpecifiedClientIP
9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
4 years, 1 week ago
Reviewed
0|i2umqf:
ZooKeeper ZOOKEEPER-2387

Attempt to gracefully stop the

Improvement Resolved Major Won't Fix Biju Nair Biju Nair Biju Nair 10/Mar/16 06:44   20/Mar/16 13:38 20/Mar/16 07:09         0 1   To stop ZooKeeper service, {{kill}} is issued against the ZK process. It will be good to gracefully stop all the subcomponents. 9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
4 years, 4 days ago 0|i2ugnz:
ZooKeeper ZOOKEEPER-2386

Cannot achieve quorum when middle server (in a q of 3) is unreacable

Bug Open Major Unresolved Unassigned Enis Soztutar Enis Soztutar 09/Mar/16 22:11   14/Sep/16 06:40           1 4   Recently, we've observed a curious case where a quorum was not reached for days in a cluster of 3 nodes (zk0, zk1, zk2) and the middle node zk1 is unreachable from network.

The leader election happens, and both zk0 and zk2 starts the vote. Then each server sends notifications to every other server including itself. The problem is that, zk1 vm is unavailable, so when we are trying to open up a socket to connect to that server with socket timeout of 5 seconds, it delays the notification processing of the vote sent from zk2 to zk2 (itself). The vote eventually comes after 5 sec, but by that time, the follower (zk0) already converted to the follower state. On the follower state, the follower try to connect to leader 5 times with 1 second timeout (5 sec in total). Since the leader does not start its peer port for 5 seconds after the follower starts, the follower always times out connecting to the leader. This cycle is repeating for hours / days even after restarting the servers several times.

I believe this is related to the default timeouts (5 sec socket timeout) and follower to leader connection timeout (5 tries with 1 second timeout). Only after setting the {{zookeeper.cnxTimeout}} to 1 second, the quorum was operating.

More logs coming shortly.

zoo.cfg:
{code}
server.3=zk2-hostname:2889:3889
server.2=zk1-hostname:2889:3889
server.1=zk0-hostname:2889:3889
{code}
9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
3 years, 27 weeks, 1 day ago 0|i2ufz3:
ZooKeeper ZOOKEEPER-2385

Zookeeper trunk build is failing on windows

Bug Closed Blocker Fixed Mohammad Arshad Mohammad Arshad Mohammad Arshad 09/Mar/16 14:45   21/Jul/16 16:18 15/Mar/16 12:17   3.4.9, 3.5.2, 3.6.0 build   0 5   command {{ant tar}} fails with following error
{code}
D:\gitHome\zookeeperTrunk\build.xml:722: The following error occurred while executing this line:
D:\gitHome\zookeeperTrunk\src\contrib\build.xml:47: The following error occurred while executing this line:
D:\gitHome\zookeeperTrunk\src\contrib\build-contrib.xml:207: Unable to delete file D:\gitHome\zookeeperTrunk\src\java\lib\ivy-2.4.0.jar
{code}

This is happening only on windows.
9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
4 years, 1 week, 2 days ago
Reviewed
0|i2uf4f:
ZooKeeper ZOOKEEPER-2384

Support atomic increment / decrement of znode value

Improvement Resolved Major Later Unassigned Ted Yu Ted Yu 09/Mar/16 12:26   11/Nov/16 19:05 11/Nov/16 19:05         0 6   Use case is to store reference count (integer type) in znode.

It is desirable to provide support for atomic increment / decrement of the znode value.

Suggestion from Flavio:
{quote}
you can read the znode, keep the version of the znode, update the value, write back conditionally. The condition for the setData operation to succeed is that the version is the same that it read
{quote}
While the above is feasible, developer has to implement retry logic him/herself. It is not easy to combine increment / decrement with other operations using multi.
atomic 9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 19 weeks, 5 days ago 0|i2ueu7:
ZooKeeper ZOOKEEPER-2383

Startup race in ZooKeeperServer

Bug Closed Blocker Fixed Rakesh Radhakrishnan Steven Rowe Steven Rowe 08/Mar/16 12:00   31/Mar/17 05:01 03/Jan/17 12:54 3.4.8 3.4.10, 3.5.3, 3.6.0 jmx, server   2 17   In attempting to upgrade Solr's ZooKeeper dependency from 3.4.6 to 3.4.8 (SOLR-8724) I ran into test failures where attempts to create a node in a newly started standalone ZooKeeperServer were failing because of an assertion in MBeanRegistry.

ZooKeeperServer.startup() first sets up its request processor chain then registers itself in JMX, but if a connection comes in before the server's JMX registration happens, registration of the connection will fail because it trips the assertion that (effectively) its parent (the server) has already registered itself.

{code:java|title=ZooKeeperServer.java}
public synchronized void startup() {
if (sessionTracker == null) {
createSessionTracker();
}
startSessionTracker();
setupRequestProcessors();

registerJMX();

state = State.RUNNING;
notifyAll();
}
{code}

{code:java|title=MBeanRegistry.java}
public void register(ZKMBeanInfo bean, ZKMBeanInfo parent)
throws JMException
{
assert bean != null;
String path = null;
if (parent != null) {
path = mapBean2Path.get(parent);
assert path != null;
}
{code}

This problem appears to be new with ZK 3.4.8 - AFAIK Solr never had this issue with ZK 3.4.6.
9223372036854775807 No Perforce job exists for this issue. 9 9223372036854775807
3 years, 8 weeks, 3 days ago 0|i2ucfr:
ZooKeeper ZOOKEEPER-2382

Make fsync.warningthresholdms property configurable through zoo.cfg

Improvement Resolved Minor Not A Problem Biju Nair Biju Nair Biju Nair 05/Mar/16 06:43   20/Mar/16 00:33 17/Mar/16 13:02         0 1   Currently {{fsync.warningthresholdms}} property can be set as a Java system property. But it would help if this property can be made configurable through {{zoo.cfg}} so that configuration management tools can leverage it. Also the Java system property name should be standardized (refer ZOOKEEPER-2316) so that the property is inline with the standard followed by other properties. 9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
4 years, 4 days ago 0|i2u8jj:
ZooKeeper ZOOKEEPER-2381

ReconfigTest.testPortChange has been failing intermittently

Test Resolved Major Duplicate Unassigned Akihiro Suda Akihiro Suda 04/Mar/16 03:35   04/Mar/16 04:09 04/Mar/16 04:09     server, tests   0 1   ReconfigTest.testPortChange has been failing intermittently:

- Feb 12, 2016: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/3045/
- Feb 16, 2015: http://permalink.gmane.org/gmane.comp.java.zookeeper.devel/25521

I can also locally reproduce with 40d0804c (Mar 3, 2016).


The error message looks like as if it is a linearizability violation for sync+read operation, but I'm still not sure.

stack trace:
{code}
junit.framework.AssertionFailedError: expected:<test[1]> but was:<test[0]>
at org.apache.zookeeper.test.ReconfigTest.testNormalOperation(ReconfigTest.java:150)
at org.apache.zookeeper.test.ReconfigTest.testPortChange(ReconfigTest.java:598)
at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:79)
{code}

test code: https://github.com/apache/zookeeper/blob/2cc945442e218d0757983ac42e2a5d86a94ccb30/src/java/test/org/apache/zookeeper/test/ReconfigTest.java#L150
{code:java}
for (int j = 0; j < 30; j++) {
try {
..
String data = "test" + j;
writer.setData("/test", data.getBytes(), -1);
reader.sync("/", null, null);
byte[] res = reader.getData("/test", null, new Stat());
Assert.assertEquals(data, new String(res));
break;
} catch (KeeperException.ConnectionLossException e) {
if (j < 29) {
Thread.sleep(1000);
} else {
// test fails if we still can't connect to the quorum after
// 30 seconds.
Assert.fail("client could not connect to reestablished quorum: giving up after 30+ seconds.");
}
}
}
{code}
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
4 years, 2 weeks, 6 days ago 0|i2u5av:
ZooKeeper ZOOKEEPER-2380

Deadlock between leader shutdown and forwarding ACK to the leader

Bug Closed Blocker Fixed Mohammad Arshad Mohammad Arshad Mohammad Arshad 04/Mar/16 00:32   21/Jul/16 16:18 23/Jun/16 17:20   3.5.2, 3.6.0 server   1 8   Zookeeper enters into deadlock while shutting down itself, thus making zookeeper service unavailable as deadlocked server is a leader. Here is the thread dump:
{code}
"QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled)" #25 prio=5 os_prio=0 tid=0x00007fbc502a6800 nid=0x834 in Object.wait() [0x00007fbc4d9a8000]      java.lang.Thread.State: WAITING (on object monitor)      at java.lang.Object.wait(Native Method)      at java.lang.Thread.join(Thread.java:1245)      - locked <
0x00000000feb78000> (a org.apache.zookeeper.server.SyncRequestProcessor)      at java.lang.Thread.join(Thread.java:1319)      at org.apache.zookeeper.server.SyncRequestProcessor.shutdown(SyncRequestProcessor.java:196)      at org.apache.zookeeper.server.quorum.ProposalRequestProcessor.shutdown(ProposalRequestProcessor.java:90)      at org.apache.zookeeper.server.PrepRequestProcessor.shutdown(PrepRequestProcessor.java:1016)      at org.apache.zookeeper.server.quorum.LeaderRequestProcessor.shutdown(LeaderRequestProcessor.java:78)      at org.apache.zookeeper.server.ZooKeeperServer.shutdown(ZooKeeperServer.java:561)      - locked <
0x00000000feb61e20> (a org.apache.zookeeper.server.quorum.LeaderZooKeeperServer)      at org.apache.zookeeper.server.quorum.QuorumZooKeeperServer.shutdown(QuorumZooKeeperServer.java:169)      - locked <
0x00000000feb61e20> (a org.apache.zookeeper.server.quorum.LeaderZooKeeperServer)      at org.apache.zookeeper.server.quorum.LeaderZooKeeperServer.shutdown(LeaderZooKeeperServer.java:102)      - locked <
0x00000000feb61e20> (a org.apache.zookeeper.server.quorum.LeaderZooKeeperServer)      at org.apache.zookeeper.server.quorum.Leader.shutdown(Leader.java:637)      at org.apache.zookeeper.server.quorum.Leader.lead(Leader.java:590)      - locked <
0x00000000feb781a0> (a org.apache.zookeeper.server.quorum.Leader)      at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1108)


"SyncThread:1" #46 prio=5 os_prio=0 tid=0x00007fbc5848f000 nid=0x867 waiting for monitor entry [0x00007fbc4ca90000]      java.lang.Thread.State: BLOCKED (on object monitor)      at org.apache.zookeeper.server.quorum.Leader.processAck(Leader.java:784)      - waiting to lock <0x00000000feb781a0> (a org.apache.zookeeper.server.quorum.Leader)      at org.apache.zookeeper.server.quorum.AckRequestProcessor.processRequest(AckRequestProcessor.java:46)      at org.apache.zookeeper.server.SyncRequestProcessor.flush(SyncRequestProcessor.java:183)      at org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:113)
{code}

Leader.lead() calls shutdown() from the synchronized block, it acquired lock on Leader.java instance
{code}
while (true) {
synchronized (this) {
long start = Time.currentElapsedTime();
.....
{code}
In the shutdown flow SyncThread is trying to acquire lock on the same Leader.java instance.

Leader thread acquired lock and waiting for SyncThread shutdown. SyncThread waiting for the lock to complete its shutdown. This is how ZooKeeper entered into deadlock
9223372036854775807 No Perforce job exists for this issue. 8 9223372036854775807
3 years, 39 weeks ago
Reviewed
0|i2u55b:
ZooKeeper ZOOKEEPER-2379

recent commit broke findbugs qabot check

Bug Closed Blocker Fixed Rakesh Radhakrishnan Patrick D. Hunt Patrick D. Hunt 03/Mar/16 10:22   21/Jul/16 16:18 04/Mar/16 19:07 3.4.9, 3.5.2, 3.6.0 3.4.9, 3.5.2, 3.6.0 build   0 5   A recent commit seems to have broken findbugs, looks like it's in ZooKeeperSaslClient

see:
https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/3075//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
4 years, 2 weeks, 5 days ago
Reviewed
0|i2u3qf:
ZooKeeper ZOOKEEPER-2378

upgrade ivy to recent version

Improvement Closed Critical Fixed Patrick D. Hunt Patrick D. Hunt Patrick D. Hunt 03/Mar/16 02:03   21/Jul/16 16:18 03/Mar/16 12:55 3.4.8, 3.5.1, 3.6.0 3.4.9, 3.5.2, 3.6.0 build   0 2   2.4.0 is the current version. 9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
4 years, 3 weeks ago
Reviewed
0|i2u2sn:
ZooKeeper ZOOKEEPER-2377

zkServer.sh should resolve canonical path from symlinks

Improvement Patch Available Minor Unresolved Siddhartha Siddhartha Siddhartha 25/Feb/16 22:09   03/Mar/16 01:54   3.4.8   scripts   0 4   Centos 6 If zkServer.sh is started from a symlink, it is not able to correctly source the other scripts because it looks in the wrong path.

Attached patch fixes this by first resolving absolute path to the script.
9223372036854775807 No Perforce job exists for this issue. 2 9223372036854775807
Patch
4 years, 3 weeks ago 0|i2tsi7:
ZooKeeper ZOOKEEPER-2376

Server fails to start if there is a zero-length TxnLog file present in the log directory

Bug Resolved Major Duplicate Unassigned Dmitry Ryabkov Dmitry Ryabkov 25/Feb/16 14:45   25/Feb/16 15:39 25/Feb/16 15:39 3.4.6   server   0 3   Windows If there is an empty TxnLog file in the log file folder, ZooKeeper server fails to start. This is the exception it logs:

2015-11-02 07:41:10.479 -0600 (,,,) main : ERROR org.apache.zookeeper.server.ZooKeeperServerMain - Unexpected exception, exiting abnormally

java.io.EOFException

at java.io.DataInputStream.readInt(DataInputStream.java:392)

at org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63)

at org.apache.zookeeper.server.persistence.FileHeader.deserialize(FileHeader.java:64)

at org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.inStreamCreated(FileTxnLog.java:576)

at org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.createInputArchive(FileTxnLog.java:595)

at org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.goToNextLog(FileTxnLog.java:561)

at org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.next(FileTxnLog.java:643)

at org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:158)

at org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:223)

at org.apache.zookeeper.server.ZooKeeperServer.loadData(ZooKeeperServer.java:272)

at org.apache.zookeeper.server.ZooKeeperServer.startdata(ZooKeeperServer.java:399)

at org.apache.zookeeper.server.NIOServerCnxnFactory.startup(NIOServerCnxnFactory.java:122)

at org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:113)

at org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:86)

at org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:52)

at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:116)

at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:78)

Zero-length log file can be created if FileTxnLog.append() crashes after it creates FileOutputStream but before it serializes and flushes the header.
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
4 years, 4 weeks ago 0|i2trs7:
ZooKeeper ZOOKEEPER-2375

Prevent multiple initialization of login object in each ZooKeeperSaslClient instance

Bug Closed Blocker Fixed yuemeng yuemeng yuemeng 25/Feb/16 04:18   21/Jul/16 16:18 02/Mar/16 01:58 3.4.6, 3.4.8, 3.5.0, 3.5.1 3.4.9, 3.5.2, 3.6.0 java client   0 8   If there are exist many ZooKeeperSaslClient instance in one process,each ZooKeeperSaslClient instance will be call synchronize method( createSaslClient),But each ZooKeeperSaslClient instance will be lock the current object(that is say ,the synchronize only for lock it's own object) ,but many instances can access the static variable login,the synchronize can't prevent other threads access the static login object,it will be cause more than one ZooKeeperSaslClient instances use the same login object,and login.startThreadIfNeeded() will be called more than one times for same login object。
it wll cause problem:
ERROR | [Executor task launch worker-1-SendThread(fi1:24002)] | Exception while trying to create SASL client: java.lang.IllegalThreadStateException | org.apache.zookeeper.client.ZooKeeperSaslClient.createSaslClient(ZooKeeperSaslClient.java:305)
9223372036854775807 No Perforce job exists for this issue. 3 9223372036854775807
4 years, 3 weeks, 1 day ago 0|i2tqi7:
ZooKeeper ZOOKEEPER-2374

Can not telnet 2181 port on aws ec2 server

Bug Resolved Blocker Cannot Reproduce Unassigned zhupengfei zhupengfei 24/Feb/16 07:28   17/Mar/16 10:32 17/Mar/16 10:32 3.4.6   server   0 2   This is the second time I faced the problem on ec2, my activemq stomp port have the same problem, but tcp message port works fine.

I have checked zookeeper.out, no error log found. And aws technical support tell it maybe caused by zookeeper.

OS Type:
Amazon Linux AMI

Network Test Result:
-bash-4.1$ netstat | grep 2181
-bash-4.1$ telnet localhost 2181
Trying 127.0.0.1...
^C
-bash-4.1$ netstat -tunpl|grep 2181
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
tcp 0 0 :::2181 :::* LISTEN 17923/java
-bash-4.1$ netstat -an |grep 2181
tcp 0 1 172.12.10.152:60171 172.12.10.152:2181 SYN_SENT
tcp 0 0 :::2181 :::* LISTEN
tcp 0 1 ::ffff:127.0.0.1:36032 ::ffff:127.0.0.1:2181 SYN_SENT
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
4 years, 1 week ago 0|i2than:
ZooKeeper ZOOKEEPER-2373

Licenses section missing from pom file

Improvement Closed Blocker Fixed Patrick D. Hunt Flavio Paiva Junqueira Flavio Paiva Junqueira 23/Feb/16 19:02   21/Jul/16 16:18 03/Mar/16 12:58   3.4.9, 3.5.2, 3.6.0     0 3   The pom file here:

https://repo1.maven.org/maven2/org/apache/zookeeper/zookeeper/3.4.8/zookeeper-3.4.8.pom

should have a section like this:

{noformat}
<licenses>
<license>
<name>The Apache Software License, Version 2.0</name>
<url>http://www.apache.org/licenses/LICENSE-2.0.txt</url>
<distribution>repo</distribution>
</license>
</licenses>
{noformat}
9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
4 years, 3 weeks ago
Reviewed
0|i2t8y7:
ZooKeeper ZOOKEEPER-2372

Add ability to specify user under which zookeeper process should be started using zkServer.sh

Improvement Open Minor Unresolved Unassigned Siddhartha Siddhartha 23/Feb/16 10:18   03/Mar/16 02:13   3.4.8   scripts   0 1   Linux Centos 6
Java 1.7
Currently the zkServer.sh script will start zookeeper as the user invoking the script.

It would be good to add the ability to specify the user (maybe in a $USER variable in conf/zookeeper-env.sh) under which the zookeeper process should be run, so that any user invoking the script do not accidentally start it as their user (esp. as root).

Addition of this feature would make zkServer.sh the only script required to manage zookeeper process.

Thanks
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
4 years, 3 weeks ago 0|i2t7yf:
ZooKeeper ZOOKEEPER-2371

zkServer.sh status does not work if JMX Port is enabled

Bug Resolved Minor Not A Bug Unassigned Siddhartha Siddhartha 23/Feb/16 10:13   25/Feb/16 20:42 25/Feb/16 20:42 3.4.8   scripts   0 1   Linux Centos 6
Java 1.7
If I try to execute 'bin/zkServer.sh status' while having
'-Dcom.sun.management.jmxremote.port=9011' in $JVMFLAGS,
zookeeper quits with "Error: Exception thrown by the agent : java.rmi.server.ExportException: Port already in use: 9011".

EIther some other means of getting status should be used, or some way of not setting JMX variables in this case should be added.

Thanks
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
4 years, 3 weeks, 6 days ago 0|i2t7xj:
ZooKeeper ZOOKEEPER-2370

Can't access Znodes after adding ACL with SASL

Bug Resolved Major Not A Problem Unassigned Chao Sun Chao Sun 23/Feb/16 02:19   24/Mar/17 12:09 23/Feb/16 06:47 3.4.5   java client   0 3   (My apology if this is not a bug.)

I'm trying to use a ZK client which has successfully authenticated with a secure ZK server using principal {{me/hostname@EXAMPLE.COM}}. However, the following simple commands failed:

{code}
[zk: hostname(CONNECTED) 0] create /zk-test "1"
Created /zk-test
[zk: hostname(CONNECTED) 1] setAcl /zk-test sasl:me/hostname@EXAMPLE.COM:cdrwa
cZxid = 0x3e3b
ctime = Mon Feb 22 23:10:36 PST 2016
mZxid = 0x3e3b
mtime = Mon Feb 22 23:10:36 PST 2016
pZxid = 0x3e3b
cversion = 0
dataVersion = 0
aclVersion = 1
ephemeralOwner = 0x0
dataLength = 3
numChildren = 0
[zk: hostname(CONNECTED) 2] getAcl /zk-test
'sasl,'me/hostname@EXAMPLE.COM
: cdrwa
[zk: hostname(CONNECTED) 3] ls /zk-test
Authentication is not valid : /zk-test
[zk: hostname(CONNECTED) 4] create /zk-test/c "2"
Authentication is not valid : /zk-test/c
{code}

I wonder what I did wrong here, or is this behavior intentional? how can I delete the znodes? Thanks.
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
2 years, 51 weeks, 6 days ago 0|i2t78f:
ZooKeeper ZOOKEEPER-2369

Flushing DataOutputStream before calling toByteArray on the underlying ByteArrayOutputStream

Bug Open Minor Unresolved emopers emopers emopers 18/Feb/16 04:13   03/Mar/16 19:14           0 1   In ./src/java/main/org/apache/zookeeper/server/ZooKeeperServer.java
{code}
ByteArrayOutputStream baos = new ByteArrayOutputStream();
BinaryOutputArchive bos = BinaryOutputArchive.getArchive(baos);
bos.writeInt(-1, "len");
rsp.serialize(bos, "connect");
if (!cnxn.isOldClient) {
bos.writeBool(
this instanceof ReadOnlyZooKeeperServer, "readOnly");
}
baos.close();
ByteBuffer bb = ByteBuffer.wrap(baos.toByteArray());
{code}

BinaryOutputArchive internally uses DataOutputStream as its stream, and when a DataOutputStream instance wraps an underlying ByteArrayOutputStream instance,
it is recommended to flush or close the DataOutputStream before invoking the underlying instances's toByteArray() . Also, it is a good practice to call flush/close explicitly as mentioned for example http://stackoverflow.com/questions/2984538/how-to-use-bytearrayoutputstream-and-dataoutputstream-simultaneously-java.
Moreover, "baos.close()" at second last line is not required as it is no-op according to [javadoc|http://docs.oracle.com/javase/7/docs/api/java/io/ByteArrayOutputStream.html]
{quote}
Closing a ByteArrayOutputStream has no effect. The methods in this class can be called after the stream has been closed without generating an IOException.
{quote}
The patch is to add flush method on "bos" before calling toByteArray on "baos". Similar behavior is also present in the following files:
./src/java/main/org/apache/zookeeper/ClientCnxn.java
./src/java/main/org/apache/zookeeper/server/ZKDatabase.java
./src/java/main/org/apache/zookeeper/server/persistence/Util.java
./src/java/main/org/apache/zookeeper/server/NIOServerCnxn.java

Let me know if this looks good. I can provide patch.
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
Patch
4 years, 3 weeks ago 0|i2szs7:
ZooKeeper ZOOKEEPER-2368

Client watches are not disconnected on close

Improvement Closed Major Fixed Timothy James Ward Timothy James Ward Timothy James Ward 15/Feb/16 04:28   20/May/19 13:50 20/Jun/18 09:23 3.4.0, 3.5.0 3.6.0, 3.5.5     0 5 0 13200   If I have a ZooKeeper client connected to an ensemble then obviously I can register watches.

If the client is disconnected (for example by a failing ensemble member) then I get a disconnection event for all of my watches. If, on the other hand, my client is closed then I *do not* get a disconnection event. This asymmetry makes it really hard to clear up properly when using the asynchronous API, as there is no way to "fail" data reads/updates when the client is closed.

I believe that the correct behaviour should be for all watchers to receive a disconnection event when the client is closed. The watchers can then respond as appropriate, and can differentiate between a "server disconnect" and a "client disconnect" by checking the ZooKeeper#getState() method.

This would not be a breaking behaviour change as Watchers are already required to handle disconnection events.
100% 100% 13200 0 pull-request-available 9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
1 year, 39 weeks, 1 day ago 0|i2sudz:
ZooKeeper ZOOKEEPER-2367

Unable to establish quorum when hostnames are not resolvable between all of the nodes

Bug Resolved Critical Duplicate Unassigned Timothy James Ward Timothy James Ward 12/Feb/16 13:52   15/Feb/16 06:59 15/Feb/16 06:59 3.5.0   quorum   0 1   If I have a set of three machines, all of which have locally defined hostnames A, B and C (i.e. B and C cannot look up A by name). I am unable to control the DNS setup, and I don't want to manually reimplement DNS using entries in the hosts file.

A is on IP 192.168.1.16
B is on IP 192.168.1.17
C is on IP 192.168.1.18

All of my ZK configuration uses literal IP addresses (no hostnames anywhere), however I still see a hostname appearing in the leader log (in this case the leader was C):

java.net.UnknownHostException: B
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:184)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:369)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.receiveConnection(QuorumCnxManager.java:291)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$Listener.run(QuorumCnxManager.java:558)

This is caused by the initiateConnection method of QuorumCnxManager, which contains the line:

self.getElectionAddress().getHostName()

The use of getHostName() forces a reverse DNS lookup, which I do not want. The code should use getHostString() instead, which will use the actual data from config, and avoid unresolvable hosts being sent over the wire. This will mean that node C attempts to connect to 192.168.1.17, not "B".
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
4 years, 5 weeks, 3 days ago 0|i2ss9z:
ZooKeeper ZOOKEEPER-2366

Reconfiguration of client port causes a socket leak

Bug Closed Blocker Fixed Flavio Paiva Junqueira Timothy James Ward Timothy James Ward 12/Feb/16 11:59   21/Jul/16 16:18 23/Jun/16 17:46 3.5.0 3.5.2, 3.6.0 quorum   0 8   The NIOServerCnxnFactory reconfigure method can leak server sockets, and hence make ports unusable until the JVM restarts:

The first line of the method takes a reference to the current ServerSocketChannel and then the next line replaces it. The subsequent interactions with the server socket can fail (for example if the reconfiguration tries to bind to an in-use port). If they fail *before* the call to oldSS.close() then oldSS is *never* closed. This holds that port open forever, and prevents the user from rolling back to the previous port!

The code from reconfigure is shown below:

ServerSocketChannel oldSS = ss;
try {
this.ss = ServerSocketChannel.open();
ss.socket().setReuseAddress(true);
LOG.info("binding to port " + addr);
ss.socket().bind(addr);
ss.configureBlocking(false);
acceptThread.setReconfiguring();
oldSS.close();
acceptThread.wakeupSelector();
try {
acceptThread.join();
} catch (InterruptedException e) {
LOG.error("Error joining old acceptThread when reconfiguring client port " + e.getMessage());
}
acceptThread = new AcceptThread(ss, addr, selectorThreads);
acceptThread.start();
} catch(IOException e) {
LOG.error("Error reconfiguring client port to " + addr + " " + e.getMessage());
}



9223372036854775807 No Perforce job exists for this issue. 8 9223372036854775807
3 years, 39 weeks ago
Reviewed
0|i2ss4v:
ZooKeeper ZOOKEEPER-2365

JAAS configuration section error is confusing

Bug Open Trivial Unresolved Biju Nair Dan Fitch Dan Fitch 10/Feb/16 17:24   05/Feb/20 07:16   3.4.6 3.7.0, 3.5.8 java client   1 6   Ubuntu x86_64 openjdk-7-jre I have zookeeper running normally just fine in a 3-server cluster.

Then I try to configure zookeeper to use Kerberos, following docs in the Solr wiki here: https://cwiki.apache.org/confluence/display/solr/Kerberos+Authentication+Plugin

I can't even get to the fun Kerberos errors. When I start with {{JVMFLAGS="-Djava.security.auth.login.config=/opt/zookeeper/jaas-server.conf"}}

and this jaas-server.conf:

{code}
Server {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
keyTab=/keytabs/vdev-solr-01.keytab
storeKey=true
doNotPrompt=true
useTicketCache=false
debug=true
principal="HTTP/<snip>";
}
{code}

I get this in the log:

{code}
2016-02-10 16:16:51,327 [myid:1] - ERROR [main:ServerCnxnFactory@195] - No JAAS configuration section named 'Server' was foundin '/opt/zookeeper/jaas-server.conf'.
2016-02-10 16:16:51,328 [myid:1] - ERROR [main:QuorumPeerMain@89] - Unexpected exception, exiting abnormally
java.io.IOException: No JAAS configuration section named 'Server' was foundin '/opt/zookeeper/jaas-server.conf'.
at org.apache.zookeeper.server.ServerCnxnFactory.configureSaslLogin(ServerCnxnFactory.java:196)
at org.apache.zookeeper.server.NIOServerCnxnFactory.configure(NIOServerCnxnFactory.java:87)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:130)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:111)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:78)
{code}

(Note the "foundin" typo.)

I get the exact same error if the jaas-server.conf file exists, or does not.

So later I found that the Solr wiki was wrong and lost the double quotes around the keytab value. It would be nice if Zookeeper spewed a more useful message when it can't parse the configuration.
9223372036854775807 No Perforce job exists for this issue. 2 9223372036854775807
1 year, 3 weeks, 2 days ago 0|i2sp3r:
ZooKeeper ZOOKEEPER-2364

"ant docs" fails on branch-3.5 due to missing releasenotes.xml.

Bug Closed Blocker Fixed Patrick D. Hunt Chris Nauroth Chris Nauroth 08/Feb/16 16:36   16/Aug/16 17:44 21/Mar/16 16:46 3.5.2 3.5.2, 3.6.0 build, documentation   0 3   "ant docs" is failing on branch-3.5. (Both trunk and branch-3.4 are fine.) The root cause appears
to be a missing file on branch-3.5: src/docs/src/documentation/content/xdocs/releasenotes.xml.
This causes Forrest to report a failure due to broken hyperlinks targeting releasenotes.html.
9223372036854775807 No Perforce job exists for this issue. 4 9223372036854775807
4 years, 3 days ago
Reviewed
0|i2skxz:
ZooKeeper ZOOKEEPER-2363

DatadirCleanupManager never created by ZookeeperServerMain

Bug Open Major Unresolved Unassigned David Foregger David Foregger 05/Feb/16 12:09   03/Aug/16 01:51   3.4.0       0 3   h4. Background
ZOOKEEPER-1107 introduced a DatadirCleanupManager to automatically purge snapshots. This can be configured using autopurge.snapRetainCount and autopurge.purgeInterval. This is documented [here|http://zookeeper.apache.org/doc/r3.4.5/zookeeperAdmin.html#Ongoing+Data+Directory+Cleanup] and [there|http://zookeeper.apache.org/doc/r3.4.5/zookeeperAdmin.html#sc_advancedConfiguration].
h4. Symptoms
Autopurging does not work when running a standalone ZooKeeperServer. The DatadirCleanupManager is started by the QuorumPeerMain, but there is no similar setup ZooKeeperServerMain. ServerConfig does not hold autopurge properties.
h4. Expected Behavior
Starting a standalone zookeeper server should enable autopurging with the same behavior as a quorum server.
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 33 weeks, 1 day ago 0|i2shfr:
ZooKeeper ZOOKEEPER-2362

ZooKeeper multi / transaction allows partial read

Bug Open Critical Unresolved Atri Sharma Whitney Sorenson Whitney Sorenson 05/Feb/16 10:32   21/Nov/18 20:09   3.4.6   server   1 9   In this thread http://mail-archives.apache.org/mod_mbox/zookeeper-user/201602.mbox/%3CCAPbqGzicBkLLyVDm7RFM20z0y3X1v1P-C9-1%3D%3D1DDqRDTzdOmQ%40mail.gmail.com%3E , I discussed an issue I've now seen in multiple environments:

In a multi (using Curator), I write 2 new nodes. At some point, I issue 2 reads for these new nodes. In one read, I see one of the new nodes. In a subsequent read, I fail to see the other new node:

1. Starting state : { /foo = <does not exist>, /bar = <does not exist> }
2. In a multi, write: { /foo = A, /bar = B}
3. Read /foo as A
4. Read /bar as <does not exist>

#3 and #4 are issued 100% sequentially.

It is not known at what point during #2, #3 starts.

Note: the reads are getChildren() calls.
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
1 year, 17 weeks ago 0|i2shbz:
ZooKeeper ZOOKEEPER-2361

Revisit 'VisibleForTesting' phrase used to indicate a member or method visible for testing

Improvement Open Minor Unresolved Rakesh Radhakrishnan Rakesh Radhakrishnan Rakesh Radhakrishnan 04/Feb/16 13:02   04/Feb/16 14:23           0 2   ZooKeeper uses {{// VisibleForTesting}} comment to indicate a member or method which is visible for unit testing. The idea of this jira is to discuss better ways to convey the message more clear and implement the same. One idea could use annotations, needs to introduce {{@VisibleForTesting}}

For example,
[ContainerManager.java#L134|https://github.com/apache/zookeeper/blob/trunk/src/java/main/org/apache/zookeeper/server/ContainerManager.java#L134], [PurgeTxnLog.java#L78|https://github.com/apache/zookeeper/blob/trunk/src/java/main/org/apache/zookeeper/server/PurgeTxnLog.java#L78], [ZooKeeper.java#L1011|https://github.com/apache/zookeeper/blob/trunk/src/java/main/org/apache/zookeeper/ZooKeeper.java#L1011] etc.

{code}
ZooKeeper.java
// VisibleForTesting
public Testable getTestable() {
return new ZooKeeperTestable(this, cnxn);
}
{code}

{code}
PurgeTxnLog.java
// VisibleForTesting
static void retainNRecentSnapshots(FileTxnSnapLog txnLog, List<File> snaps) {
{code}
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
4 years, 7 weeks ago 0|i2sfrj:
ZooKeeper ZOOKEEPER-2360

Update commons collections version used by tests/releaseaudit

Bug Closed Blocker Fixed Patrick D. Hunt Patrick D. Hunt Patrick D. Hunt 04/Feb/16 12:33   21/Jul/16 16:18 04/Feb/16 19:33 3.4.7, 3.5.1 3.4.8, 3.5.2 build   0 4   I don't believe this affects us from a security perspective directly, however it's something we should clean up in our next release.

Afaict the only commons we use for shipping/production code is commons-cli. Our two release branches, 3.4 and 3.5, neither of them use commons-collections. I looked at the binary release artifact and it doesn't include the commons collections jar.

We do have a test that uses CollectionsUtils, but no shipping code. I downloaded our 3.4 and 3.5 artifacts, this is all I see:

phunt:~/Downloads/zd/5/zookeeper-3.5.1-alpha$ grep -R "org.apache.commons.collections" .
./src/java/test/org/apache/zookeeper/RemoveWatchesTest.java:import org.apache.commons.collections.CollectionUtils;
phunt:~/Downloads/zd/5/zookeeper-3.5.1-alpha$

Also in our ivy file we have

<dependency org="org.apache.rat" name="apache-rat-tasks"
rev="0.10" conf="releaseaudit->default"/>
<dependency org="commons-lang" name="commons-lang"
rev="2.6" conf="releaseaudit->default"/>
<dependency org="commons-collections" name="commons-collections"
rev="3.2.1" conf="releaseaudit->default"/>

So commons-collections is pulled in - but only for the release audit, which is something we do as a build verification activity but not part of the product itself.
9223372036854775807 No Perforce job exists for this issue. 3 9223372036854775807
4 years, 6 weeks, 6 days ago
Reviewed
0|i2sfo7:
ZooKeeper ZOOKEEPER-2359

ZooKeeper client has unnecessary logs for watcher removal errors

Improvement Resolved Major Fixed Jordan Zimmerman Jordan Zimmerman Jordan Zimmerman 04/Feb/16 08:30   27/May/19 04:17 01/Jun/17 12:38 3.5.1 3.5.4, 3.6.0 java client   8 11   ClientCnxn.java logs errors during watcher removal:

LOG.error("Failed to find watcher!", nwe);
LOG.error("Exception when removing watcher", ke);

An error code/exception is generated so the logs are noisy and unnecessary. If the client handles the error there's still a log message. This is different than other APIs. These logs should be removed.
9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
42 weeks, 3 days ago 0|i2sf8n:
ZooKeeper ZOOKEEPER-2358

NettyServerCnxn leaks watches upon close

Bug Open Major Unresolved Ian Dimayuga Ian Dimayuga Ian Dimayuga 27/Jan/16 21:35   05/Feb/20 07:16   3.4.7, 3.5.1 3.7.0, 3.5.8     0 6   NettyServerCnxn.close() neglects to call zkServer.removeCnxn the way NIOServerCnxn.close() does. Also, WatchLeakTest does not test watch leaks in Netty. 9223372036854775807 No Perforce job exists for this issue. 2 9223372036854775807
3 years, 31 weeks, 2 days ago 0|i2s2jr:
ZooKeeper ZOOKEEPER-2357

Unhandled errors propagating through cluster

Task Open Minor Unresolved Unassigned Gareth Humphries Gareth Humphries 22/Jan/16 11:26   05/Nov/17 15:32   3.4.6   leaderElection, quorum, server   1 5   Hi,

I need some help understanding a recurring problem we're seeing with our zookeeper cluster. It's a five node cluster that ordinarily runs fine. Occasionally we see an error from which the cluster recovers, but it causes a lot of grief and I'm sure is representative of an unhealthy situation.

To my eye it looks like an invalid bit of data getting into the system and not being handled gracefully; I'm the first to say my eye is not expert though, so I humbly submit an annotated log exert in the hope some who knows more than me can provide some illumination.


The cluster seems to be ticking along fine, until we get errors on 2 of the 5 nodes like so:
2016-01-19 13:12:49,698 - WARN [QuorumPeer[myid=1]/0.0.0.0:2181:Follower@89] - Exception when following the leader
java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:392)
at org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63)
at org.apache.zookeeper.server.quorum.QuorumPacket.deserialize(QuorumPacket.java:83)
at org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:103)
at org.apache.zookeeper.server.quorum.Learner.readPacket(Learner.java:153)
at org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:85)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:786)
2016-01-19 13:12:49,698 - INFO [QuorumPeer[myid=1]/0.0.0.0:2181:Follower@166] - shutdown called
java.lang.Exception: shutdown Follower
at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:166)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:790)

This is immediately followed by 380 occurences of:
2016-01-19 13:12:49,699 - INFO [QuorumPeer[myid=1]/0.0.0.0:2181:NIOServerCnxn@1007] - Closed socket connection for client /X.Y.Z.56:59028 which had sessionid 0x151b01ee8330234

and a:
2016-01-19 13:12:49,766 - INFO [QuorumPeer[myid=1]/0.0.0.0:2181:FollowerZooKeeperServer@139] - Shutting down
2016-01-19 13:12:49,766 - INFO [QuorumPeer[myid=1]/0.0.0.0:2181:ZooKeeperServer@441] - shutting down
2016-01-19 13:12:49,766 - INFO [QuorumPeer[myid=1]/0.0.0.0:2181:FollowerRequestProcessor@105] - Shutting down
2016-01-19 13:12:49,766 - INFO [QuorumPeer[myid=1]/0.0.0.0:2181:CommitProcessor@181] - Shutting down
2016-01-19 13:12:49,766 - INFO [QuorumPeer[myid=1]/0.0.0.0:2181:FinalRequestProcessor@415] - shutdown of request processor complete
2016-01-19 13:12:49,767 - INFO [QuorumPeer[myid=1]/0.0.0.0:2181:SyncRequestProcessor@209] - Shutting down
2016-01-19 13:12:49,767 - INFO [CommitProcessor:1:CommitProcessor@150] - CommitProcessor exited loop!
2016-01-19 13:12:49,767 - INFO [FollowerRequestProcessor:1:FollowerRequestProcessor@95] - FollowerRequestProcessor exited loop!
2016-01-19 13:13:09,418 - WARN [SyncThread:1:FileTxnLog@334] - fsync-ing the write ahead log in SyncThread:1 took 30334ms which will adversely effect operation latency. See the ZooKeeper troubleshooting guide
2016-01-19 13:13:09,427 - WARN [SyncThread:1:SendAckRequestProcessor@64] - Closing connection to leader, exception during packet send
java.net.SocketException: Socket closed
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:121)
at java.net.SocketOutputStream.write(SocketOutputStream.java:159)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
at org.apache.zookeeper.server.quorum.Learner.writePacket(Learner.java:139)
at org.apache.zookeeper.server.quorum.SendAckRequestProcessor.flush(SendAckRequestProcessor.java:62)
at org.apache.zookeeper.server.SyncRequestProcessor.flush(SyncRequestProcessor.java:204)
at org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:131)
2016-01-19 13:13:09,428 - INFO [SyncThread:1:SyncRequestProcessor@187] - SyncRequestProcessor exited!


As a small aside, the fsync log errors for the first two servers to be hit are:
2016-01-19 13:13:09,418 - WARN [SyncThread:1:FileTxnLog@334] - fsync-ing the write ahead log in SyncThread:1
took 30334ms which will adversely effect operation latency.
2016-01-19 13:13:09,539 - WARN [SyncThread:2:FileTxnLog@334] - fsync-ing the write ahead log in SyncThread:2
took 30456ms which will adversely effect operation latency.

If rewind from date of the entry the milliseconds given, you arrive within one millisecond of the same time on each server.
But I digress.

For the next 12 minutes or so, the logs are full of the below sort of exceptions, in seemingly no consistent order or frequency:
2016-01-19 13:13:09,440 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2016-01-19 13:13:09,441 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007] - Closed socket connection for client /X.Y.Z.181:51381 (no session established for client)
2016-01-19 13:13:09,443 - WARN [QuorumPeer[myid=1]/0.0.0.0:2181:SendAckRequestProcessor@64] - Closing connection to leader, exception during packet send
java.net.SocketException: Socket closed
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:121)
at java.net.SocketOutputStream.write(SocketOutputStream.java:159)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
at org.apache.zookeeper.server.quorum.Learner.writePacket(Learner.java:139)
at org.apache.zookeeper.server.quorum.SendAckRequestProcessor.flush(SendAckRequestProcessor.java:62)
at org.apache.zookeeper.server.SyncRequestProcessor.flush(SyncRequestProcessor.java:204)
at org.apache.zookeeper.server.SyncRequestProcessor.shutdown(SyncRequestProcessor.java:216)
at org.apache.zookeeper.server.quorum.FollowerZooKeeperServer.shutdown(FollowerZooKeeperServer.java:147)
at org.apache.zookeeper.server.quorum.Learner.shutdown(Learner.java:546)
at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:167)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:790)
2016-01-19 13:13:09,443 - INFO [QuorumPeer[myid=1]/0.0.0.0:2181:QuorumPeer@714] - LOOKING
2016-01-19 13:13:11,782 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@357] - caught end of stream exception
EndOfStreamException: Unable to read additional data from client sessionid 0x1525a047dc20005, likely client has closed socket
at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
at java.lang.Thread.run(Thread.java:744)
2016-01-19 13:13:11,783 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007] - Closed socket connection for client /X.Y.Z.1:59576 which had sessionid 0x1525a047dc20005
2016-01-19 13:13:11,784 - ERROR [CommitProcessor:1:NIOServerCnxn@178] - Unexpected Exception:
java.nio.channels.CancelledKeyException
at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73)
at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:77)
at org.apache.zookeeper.server.NIOServerCnxn.sendBuffer(NIOServerCnxn.java:151)
at org.apache.zookeeper.server.NIOServerCnxn.sendResponse(NIOServerCnxn.java:1081)
at org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:404)
at org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:74)
2016-01-19 13:25:43,898 - INFO [WorkerReceiver[myid=1]:FastLeaderElection@597] - Notification: 1 (message format version), 2 (n.leader), 0x2a001d352d (n.zxid), 0xb (n.round), LOOKING (n.state), 2 (n.sid), 0x2a (n.peerEpoch) FOLLOWING (my state)
2016-01-19 13:25:43,901 - WARN [QuorumPeer[myid=1]/0.0.0.0:2181:Follower@89] - Exception when following the leader
java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at java.io.BufferedInputStream.read(BufferedInputStream.java:254)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63)
at org.apache.zookeeper.server.quorum.QuorumPacket.deserialize(QuorumPacket.java:83)
at org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:103)
at org.apache.zookeeper.server.quorum.Learner.readPacket(Learner.java:153)
at org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:85)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:786)
2016-01-19 13:25:43,901 - INFO [QuorumPeer[myid=1]/0.0.0.0:2181:Follower@166] - shutdown called
java.lang.Exception: shutdown Follower
at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:166)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:790)



Until eventually we get to:

2016-01-19 13:26:05,099 - INFO [QuorumPeer[myid=1]/0.0.0.0:2181:QuorumPeer@784] - FOLLOWING
2016-01-19 13:26:05,099 - INFO [QuorumPeer[myid=1]/0.0.0.0:2181:ZooKeeperServer@162] - Created server with tickTime 2000 minSessionTimeout 4000
maxSessionTimeout 40000 datadir /var/lib/zookeeper_1/data/version-2 snapdir /var/lib/zookeeper_1/data/version-2
2016-01-19 13:26:05,099 - INFO [QuorumPeer[myid=1]/0.0.0.0:2181:Follower@63] - FOLLOWING - LEADER ELECTION TOOK - 21179
2016-01-19 13:26:05,100 - WARN [QuorumPeer[myid=1]/0.0.0.0:2181:Learner@233] - Unexpected exception, tries=0, connecting to zoo005/X.Y.Z.71:2888
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.server.quorum.Learner.connectToLeader(Learner.java:225)
at org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:71)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:786)



And things start to come right. Right about now, the three member which had so far escaped begin to exhibit the same behaviour.

Again, if we look at the fsync messages:
2016-01-19 13:26:06,192 - WARN [SyncThread:3:FileTxnLog@334] - fsync-ing the write ahead log in SyncThread:3
took 51394ms which will adversely effect operation latency.
2016-01-19 13:26:05,960 - WARN [SyncThread:4:FileTxnLog@334] - fsync-ing the write ahead log in SyncThread:4
took 51162ms which will adversely effect operation latency.
2016-01-19 13:26:04,524 - WARN [SyncThread:5:FileTxnLog@334] - fsync-ing the write ahead log in SyncThread:5
took 49726ms which will adversely effect operation latency.

And we rewind the number of milliseconds from the log entry timestamp, we arrive at exactly 13:25:14,798 for all three events.


So, it looks for all the world like something entered the system at 13:12:39,084, caused havoc on two nodes for 12.5 minutes, then at 13:25:14,798 it got off those and made to the other three, where it again caused havoc, before things eventually recovered and the world kept on ticking, only a medium sized log explosion worse for it.

There is nothing in any of the logs within a second of either of those times.


I'm hoping someone familiar with the code can look at those stack traces and understand what might cause such an incident.

I'm to help anyway I can. I have more complete logs, and we see this every couple of weeks or so, so can setup some additional logging if it would be of value. Let me know.
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
2 years, 19 weeks, 4 days ago 0|i2ruov:
ZooKeeper ZOOKEEPER-2356

SaslServer don't keep user information.Cant find who create the znode.

Improvement Open Major Unresolved Unassigned caixiaofeng caixiaofeng 20/Jan/16 21:12   20/Jan/16 21:12   3.5.1   server   0 5   kerberos server as zookeper data only keep data like this,will not keep any user information for create user。 then don't know who create this znode in Sasl Auth env。 hope zookeper can store create/modify user information,such as other file system.。

[zk: x.x.x.x:24002(CONNECTED) 2] stat /t2
cZxid = 0x10001e409
ctime = Tue Dec 22 17:51:50 CST 2015
mZxid = 0x10001e409
mtime = Tue Dec 22 17:51:50 CST 2015
pZxid = 0x10001e409
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 7
numChildren = 0
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
4 years, 9 weeks ago 0|i2rrz3:
ZooKeeper ZOOKEEPER-2355

Ephemeral node is never deleted if follower fails while reading the proposal packet

Bug Resolved Critical Fixed Mohammad Arshad Mohammad Arshad Mohammad Arshad 18/Jan/16 09:54   13/Nov/18 08:48 03/Aug/17 11:58 3.4.8, 3.4.9, 3.4.10, 3.5.1, 3.5.2, 3.5.3 3.4.11, 3.5.4, 3.6.0 quorum, server   12 21   ZOOKEEPER-2834 ZooKeeper ephemeral node is never deleted if follower fail while reading the proposal packet
The scenario is as follows:
# Configure three node ZooKeeper cluster, lets say nodes are A, B and C, start all, assume A is leader, B and C are follower
# Connect to any of the server and create ephemeral node /e1
# Close the session, ephemeral node /e1 will go for deletion
# While receiving delete proposal make Follower B to fail with {{SocketTimeoutException}}. This we need to do to reproduce the scenario otherwise in production environment it happens because of network fault.
# Remove the fault, just check that faulted Follower is now connected with quorum
# Connect to any of the server, create the same ephemeral node /e1, created is success.
# Close the session, ephemeral node /e1 will go for deletion
# {color:red}/e1 is not deleted from the faulted Follower B, It should have been deleted as it was again created with another session{color}
# {color:green}/e1 is deleted from Leader A and other Follower C{color}
9223372036854775807 No Perforce job exists for this issue. 5 9223372036854775807
2 years, 33 weeks ago 0|i2rn73:
ZooKeeper ZOOKEEPER-2354

ZOOKEEPER-1653 not merged in master and 3.5 branch

Bug Patch Available Major Unresolved Sujith Simon Mohammad Arshad Mohammad Arshad 12/Jan/16 10:20   05/Feb/20 07:12     3.7.0, 3.5.8     3 6 0 7200   ZOOKEEPER-1653 is merged only to 3.4 branch.
It should be merged to 3.5 and master branch as well.
100% 100% 7200 0 pull-request-available 9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
24 weeks, 2 days ago 0|i2r6sf:
ZooKeeper ZOOKEEPER-2353

QuorumCnxManager protocol needs to be upgradable with-in a specific Version

Improvement Open Major Unresolved Unassigned Powell Molleti Powell Molleti 11/Jan/16 14:54   18/Jan/16 19:36   3.4.7, 3.5.1       0 7   Currently 3.5.X sends its hdr as follows:

{code:title=QuorumCnxManager.java|borderStyle=solid}
dout.writeLong(PROTOCOL_VERSION);
dout.writeLong(self.getId());
String addr = self.getElectionAddress().getHostString() + ":" + self.getElectionAddress().getPort();
byte[] addr_bytes = addr.getBytes();
dout.writeInt(addr_bytes.length);
dout.write(addr_bytes);
dout.flush();
{code}

Since it writes length of host and port byte string there is no simple way to append new fields to this hdr anymore. I.e the rx side has to consider all bytes after sid for host and port parsing, which is what it does here:

[QuorumCnxManager.InitialMessage.parse(): http://bit.ly/1Q0znpW]
{code:title=QuorumCnxManager.java|borderStyle=solid}
sid = din.readLong();

int remaining = din.readInt();
if (remaining <= 0 || remaining > maxBuffer) {
throw new InitialMessageException(
"Unreasonable buffer length: %s", remaining);
}

byte[] b = new byte[remaining];
int num_read = din.read(b);

if (num_read != remaining) {
throw new InitialMessageException(
"Read only %s bytes out of %s sent by server %s",
num_read, remaining, sid);
}

// FIXME: IPv6 is not supported. Using something like Guava's HostAndPort
// parser would be good.
String addr = new String(b);
String[] host_port = addr.split(":");
{code}

This has been captured in the discussion here: ZOOKEEPER-2186.
Though it is possible to circumvent this problem by various means the request here is to design messages with hdr such that there is no need to bump version number or hack certain fields (i.e figure out if its length of host/port or length of different message etc, in the above case).

This is the idea here as captured in ZOOKEEPER-2186.
{code:java}
dout.writeLong(PROTOCOL_VERSION);

String addr = self.getElectionAddress().getHostString() + ":" + self.getElectionAddress().getPort();
byte[] addr_bytes = addr.getBytes();

// After version write the total length of msg sent by sender.
dout.writeInt(Long.BYTES + addr_bytes.length);
// Write sid afterwards
dout.writeLong(self.getId());
// Write length of host/port string
dout.writeInt(addr_bytes.length);
// Write host/port string
dout.write(addr_bytes);
{code}

Since total length of the message and length of each variable field is also present it is quite easy to provide backward compatibility, w.r.t to parsing of the message.
Older code will read the length of message it knows and ignore the rest. Newer revision(s), that wants to keep things compatible, will only append to hdr and not change the meaning of current fields.

I am guessing this was the original intent w.r.t the introduction of protocol version here: ZOOKEEPER-1633

Since 3.4.x code does not parse this and 3.5.x is still in alpha mode perhaps it is possible to consider this change now?.

Also I would like to propose to carefully consider the option of using protobufs for the next protocol version bump. This will prevent issues like this in the future.

9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
4 years, 9 weeks, 2 days ago 0|i2r593:
ZooKeeper ZOOKEEPER-2352

rpm build broke

Bug Resolved Major Won't Fix Unassigned Wenjie Ding Wenjie Ding 05/Jan/16 15:14   03/Mar/16 11:20 03/Mar/16 11:20 3.5.1   build   0 3 36000 36000 0% ubuntu 15.10 rpm ‘cd’ to the BUILD directory, delete the directory it just ‘cd’ to. What was going on from that point was that it called some script which was loading /bin/bash and crashed since ‘getcwd’ cannot access the parent directory which had been deleted.

build output messages;
...
[rpm] + cd /tmp/zkpython_build_root/BUILD
[rpm] + '[' /tmp/zkpython_build_root/BUILD '!=' / ']'
[rpm] + rm -rf /tmp/zkpython_build_root/BUILD
[rpm] ++ dirname /tmp/zkpython_build_root/BUILD
[rpm] + mkdir -p /tmp/zkpython_build_root
[rpm] + mkdir /tmp/zkpython_build_root/BUILD
[rpm] + /usr/lib/rpm/check-buildroot
[rpm] shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
[rpm] + /usr/lib/rpm/redhat/brp-compress
[rpm] shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
[rpm] chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
[rpm] + /usr/lib/rpm/redhat/brp-strip /usr/bin/strip
[rpm] shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
[rpm] + /usr/lib/rpm/redhat/brp-strip-static-archive /usr/bin/strip
[rpm] shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
[rpm] + /usr/lib/rpm/redhat/brp-strip-comment-note /usr/bin/strip /usr/bin/objdump
[rpm] shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
[rpm] + /usr/lib/rpm/brp-python-bytecompile
[rpm] shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
0% 0% 36000 36000 9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
4 years, 3 weeks ago 0|i2qthz:
ZooKeeper ZOOKEEPER-2351

%JAVA_HOME% in bin\zkEnv.cmd does not work on Windows 8 and Windows 10

Bug Resolved Major Duplicate Unassigned LEON YU LEON YU 03/Jan/16 22:34   04/Jan/16 18:02 04/Jan/16 18:02 3.4.7   scripts   0 2   %JAVA_HOME% does not work in zkEnv.cmd, so zkServer.cmd and zkClient.cmd can not run and zookeeper can not start.

Temporary solution to this is to use quotation marks "%JAVA_HOME%" to replace %JAVA_HOME% in zkEnv.cmd.
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
4 years, 11 weeks, 3 days ago
Incompatible change
0|i2qqi7:
ZooKeeper ZOOKEEPER-2350

Script that provide a way to build the ensemble with ease.

New Feature Open Minor Unresolved Unassigned Minoru Osuka Minoru Osuka 27/Dec/15 20:33   29/Feb/16 00:31   3.5.0   scripts   1 3   Starting with 3.5.0, ZooKeeper has already supported a very convenient dynamic reconfiguration. However, the procedure is slightly complicated. So I propose adding a script to build ensemble with ease.

Usage:
{noformat}
$ ./bin/zkEnsemble.sh help
usage: ./bin/zkEnsemble.sh {start|stop|status} <parameters>
Commadns:
start Start a node of ensemble.
Parameters:
--seed Specify the IP address and port of an existing ensemble node that required for 2nd and subsequent nodes.
This is not required for the 1st node. (Example: 127.0.0.1:2181)
--ip Normally, you do not need to specify because it is automatically detected.
If it seems the wrong IP address is found automatically, you can over ride the IP address with this option.
--clientport The port is used to client connections (2181 by default).
--peerport The port is used to talk to each other (2888 by default).
If omitted, it will use the minimum port number that is available between 2888 to 3142.
--electionport The port is used to leader election (3888 by default).
If omitted, it will use the minimum port number that is available between 3888 to 4142.
--role The role of node, it can be participant or observer (participant by default).
--clientip The IP address for client connections (0.0.0.0 by default).
If omitted, it will use the minimum port number that is available between 2181 to 2435.
--confdir Specify a base conf directory (/Users/mosuka/git/zookeeper/conf by default).
--datadir Specify a base data directory (/tmp/zookeeper by default).
--foreground Start node in foreground.

stop Stop a node of ensemble.
Parameters:
--ip Normally, you do not need to specify because it is automatically detected.
If it seems the wrong IP address is found automatically, you can over ride the IP address with this option.
--clientport The port is used to client connections (2181 by default).

status Show ensemble nodes.
Parameters:
--seed Specify the IP address and port of a existing ensemble node (Example: 127.0.0.1:2181).

help Display this message.
{noformat}

Example:
1. Start a 1st node of ensemble on host1(192.168.33.11)
{noformat}
$ ./bin/zkEnsemble.sh start
ZooKeeper JMX enabled by default
Using config: /Users/minoru/zookeeper/zookeeper-3.5.0/conf/server.1.cfg
Starting zookeeper ... STARTED
{noformat}

2. Start a 2nd node of ensemble on host2(192.168.33.12).
{noformat}
$ ./bin/zkEnsemble.sh start --seed=192.168.33.11:2181
ZooKeeper JMX enabled by default
Using config: /Users/minoru/zookeeper/zookeeper-3.5.0/conf/server.2.cfg
Starting zookeeper ... STARTED
{noformat}

3. Start a 3rd node of ensemble on host3(192.168.33.13).
{noformat}
$ ./bin/zkEnsemble.sh start --seed=192.168.33.11:2181
ZooKeeper JMX enabled by default
Using config: /Users/minoru/zookeeper/zookeeper-3.5.0/conf/server.3.cfg
Starting zookeeper ... STARTED
{noformat}

4. Show ensemble nodes on host1(192.168.33.11).
{noformat}
$ ./bin/zkEnsemble.sh status --seed=192.168.33.11:2181
server.1=192.168.33.11:2888:3888:participant;0.0.0.0:2181
server.2=192.168.33.12:2888:3888:participant;0.0.0.0:2181
server.3=192.168.33.13:2888:3888:participant;0.0.0.0:2181
{noformat}

5. Stop a 2nd node of ensemble on host2(192.168.33.12).
{noformat}
$ ./bin/zkEnsemble.sh stop
Using config: /Users/minoru/zookeeper/zookeeper-3.5.0/conf/server.2.cfg
Stopping zookeeper ... STOPPED
{noformat}

6. Show ensemble nodes on host1(192.168.33.11).
{noformat}
$ ./bin/zkEnsemble.sh status --seed=192.168.33.11:2181
server.1=192.168.33.11:2888:3888:participant;0.0.0.0:2181
server.3=192.168.33.13:2888:3888:participant;0.0.0.0:2181
{noformat}

7. Start a 2nd node of ensemble on host2(192.168.33.12).
{noformat}
$ ./bin/zkEnsemble.sh start --seed=192.168.33.11:2181
ZooKeeper JMX enabled by default
Using config: /Users/minoru/zookeeper/zookeeper-3.5.0/conf/server.2.cfg
Starting zookeeper ... STARTED
{noformat}

8. Show ensemble nodes on host1(192.168.33.11).
{noformat}
$ ./bin/zkEnsemble.sh status --seed=192.168.33.11:2181
server.1=192.168.33.11:2888:3888:participant;0.0.0.0:2181
server.2=192.168.33.12:2888:3888:participant;0.0.0.0:2181
server.3=192.168.33.13:2888:3888:participant;0.0.0.0:2181
{noformat}
9223372036854775807 No Perforce job exists for this issue. 8 9223372036854775807
4 years, 3 weeks, 3 days ago 0|i2qbxb:
ZooKeeper ZOOKEEPER-2349

Update documentation for snapCount

Bug Resolved Minor Fixed maoling Raghavendra Prabhu Raghavendra Prabhu 21/Dec/15 09:17   11/Sep/17 17:37 11/Sep/17 16:43   3.4.11, 3.5.4, 3.6.0 documentation   0 7   The documentation states that

{code}
ZooKeeper logs transactions to a transaction
log. After snapCount transactions are written to a log
file a snapshot is started and a new transaction log
file is created. The default snapCount is
100,000.
{code}

However, in implementation, snapshotting is done when logCount is somwhere in (snapCount/2, snapCount+1], based on the limit set at runtime:

{code}
if (logCount > (snapCount / 2 + randRoll)) {
{code}
as in

https://github.com/apache/zookeeper/blob/trunk/src/java/main/org/apache/zookeeper/server/SyncRequestProcessor.java#L124
9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
2 years, 27 weeks, 3 days ago 0|i2q5sn:
ZooKeeper ZOOKEEPER-2348

Data between leader and followers are not synchronized.

Bug Open Major Unresolved Unassigned Echo Chen Echo Chen 18/Dec/15 05:17   30/Apr/19 00:56   3.5.1       0 9   When client session expired, leader tried to remove it from session map and remove its EPHEMERAL znode, for example, /test_znode. This operation succeed on leader, but at the very same time, network fault happended and not synced to followers, a new leader election launched. After leader election finished, the new leader is not the old leader. we found the znode /test_znode still existed in the followers but not on leader

*Scenario :*
1) Create znode E.g.
{{/rmstore/ZKRMStateRoot/RMAppRoot/application_1449644945944_0001/appattempt_1449644945944_0001_000001}}
2) Delete Znode.
3) Network fault b/w follower and leader machines
4) leader election again and follower became leader.

Now data is not synced with new leader..After this client is not able to same znode.

9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
46 weeks, 2 days ago 0|i2q32n:
ZooKeeper ZOOKEEPER-2347

Deadlock shutting down zookeeper

Bug Resolved Blocker Fixed Rakesh Radhakrishnan Ted Yu Ted Yu 16/Dec/15 16:01   14/Jan/16 00:22 14/Jan/16 00:22 3.4.7 3.4.8     0 10   HBase recently upgraded to zookeeper 3.4.7

In one of the tests, TestSplitLogManager, there is reproducible hang at the end of the test.
Below is snippet from stack trace related to zookeeper:
{code}
"main-EventThread" daemon prio=5 tid=0x00007fd27488a800 nid=0x6f1f waiting on condition [0x000000011834b000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007c5b8d3a0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501)

"main-SendThread(localhost:59510)" daemon prio=5 tid=0x00007fd274eb4000 nid=0x9513 waiting on condition [0x0000000118042000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060)

"SyncThread:0" prio=5 tid=0x00007fd274d02000 nid=0x730f waiting for monitor entry [0x00000001170ac000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.apache.zookeeper.server.ZooKeeperServer.decInProcess(ZooKeeperServer.java:512)
- waiting to lock <0x00000007c5b62128> (a org.apache.zookeeper.server.ZooKeeperServer)
at org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:144)
at org.apache.zookeeper.server.SyncRequestProcessor.flush(SyncRequestProcessor.java:200)
at org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:131)

"main-EventThread" daemon prio=5 tid=0x00007fd2753a3800 nid=0x711b waiting on condition [0x0000000117a30000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007c9b106b8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501)

"main" prio=5 tid=0x00007fd276000000 nid=0x1903 in Object.wait() [0x0000000108aa1000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000007c5b66400> (a org.apache.zookeeper.server.SyncRequestProcessor)
at java.lang.Thread.join(Thread.java:1281)
- locked <0x00000007c5b66400> (a org.apache.zookeeper.server.SyncRequestProcessor)
at java.lang.Thread.join(Thread.java:1355)
at org.apache.zookeeper.server.SyncRequestProcessor.shutdown(SyncRequestProcessor.java:213)
at org.apache.zookeeper.server.PrepRequestProcessor.shutdown(PrepRequestProcessor.java:770)
at org.apache.zookeeper.server.ZooKeeperServer.shutdown(ZooKeeperServer.java:478)
- locked <0x00000007c5b62128> (a org.apache.zookeeper.server.ZooKeeperServer)
at org.apache.zookeeper.server.NIOServerCnxnFactory.shutdown(NIOServerCnxnFactory.java:266)
at org.apache.hadoop.hbase.zookeeper.MiniZooKeeperCluster.shutdown(MiniZooKeeperCluster.java:301)
{code}
Note the address (0x00000007c5b66400) in the last hunk which seems to indicate some form of deadlock.

According to Camille Fournier:

We made shutdown synchronized. But decrementing the requests is
also synchronized and called from a different thread. So yeah, deadlock.
This came in with ZOOKEEPER-1907
9223372036854775807 No Perforce job exists for this issue. 5 9223372036854775807
4 years, 10 weeks ago 0|i2q00v:
ZooKeeper ZOOKEEPER-2346

SASL Auth failure manifested to client as connection refusal

Bug Patch Available Major Unresolved Meyer Kizner Steve Loughran Steve Loughran 15/Dec/15 12:59   28/Nov/16 14:44   3.4.6   server   0 5   If a client can't authenticate via sasl then (a) the stack trace is lost on the server logs, and (b) it is exposed to the client as a connection refusal. This results in curator retrying many times before giving up —and with the cause being misinterpreted as a server-down problem, rather than a client-not-trusted problem 9223372036854775807 No Perforce job exists for this issue. 2 9223372036854775807
3 years, 16 weeks, 3 days ago 0|i2pxfb:
ZooKeeper ZOOKEEPER-2345

ZOOKEEPER-2344 ServerCnxnFactory.configureSaslLogin() loses stack trace on auth failures

Sub-task Open Major Unresolved Unassigned Steve Loughran Steve Loughran 15/Dec/15 12:50   22/Mar/16 06:01   3.4.6   server   0 6   When there's a problem authenticating in {{ServerCnxnFactory.configureSaslLogin() }}, the exception is retained -but the full stack lost.
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
4 years, 6 weeks, 6 days ago 0|i2pxdr:
ZooKeeper ZOOKEEPER-2344

Provide more diagnostics/stack traces on SASL Auth failure

Improvement Open Major Unresolved Unassigned Steve Loughran Steve Loughran 15/Dec/15 12:47   22/Jun/18 00:49   3.4.7, 3.5.1, 3.4.11   java client, server   0 8   ZOOKEEPER-2345 When Kerberos decides it doesn't want to work, the JRE libraries provide some terse and unhelpful error messages.

The only way to debug the problem is (a) to have complete stack traces and (b) as much related information as possible.

Zookeeper could do more here. Currently too much of the code loses stack traces; sometimes auth errors aren't reported back to the client (the connection is closed) +others

Everyone who has tried to diagnose kerberos problems will appreciate improvements here
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 31 weeks, 2 days ago 0|i2pxcv:
ZooKeeper ZOOKEEPER-2343

Zookeeper 3.5.1 failed to deploy into the kubernetes

Bug Open Major Unresolved Unassigned cheyang cheyang 15/Dec/15 06:06   30/Nov/17 04:27   3.5.1 3.5.1 quorum   0 5   CentOS Linux release 7.1.1503 (Core)
openjdk version "1.8.0_65"
OpenJDK Runtime Environment (build 1.8.0_65-b17)
OpenJDK 64-Bit Server VM (build 25.65-b01, mixed mode)
zookeeper: 3.5.1-alpha-1693007
I'd like to setup 3-nodes zookeeper cluster with version 3.5.1. Because the network model of kubernetes is that pod and service has different ip address. In order to deploy it into kubernetes, I have to make zookeeper pods itself to be 0.0.0.0, so it can be started correctly. the configuration as below:

zk1: zoo.cfg

standaloneEnabled=false
dynamicConfigFile=/opt/zookeeper/conf/zoo.cfg.dynamic

zoo.cfg.dynamic

server.1=0.0.0.0:2888:3888:participant;2181
server.2=10.62.56.192:2888:3888:participant;2181
server.3=10.62.56.193:2888:3888:participant;2181

zk2: zoo.cfg

standaloneEnabled=false
dynamicConfigFile=/opt/zookeeper/conf/zoo.cfg.dynamic

zoo.cfg.dynamic

server.1=10.62.56.191:2888:3888:participant;2181
server.2=0.0.0.0:2888:3888:participant;2181
server.3=10.62.56.193:2888:3888:participant;2181

zk3: zoo.cfg

standaloneEnabled=false
dynamicConfigFile=/opt/zookeeper/conf/zoo.cfg.dynamic

zoo.cfg.dynamic

server.1=10.62.56.191:2888:3888:participant;2181
server.2=10.62.56.192:2888:3888:participant;2181
server.3=0.0.0.0:2888:3888:participant;218

The result is that:
1. Looks like the election is successful. a new dynamic file is generated in every node:/opt/zookeeper/conf/zoo.cfg.dynamic.100000000
like below
server.1=10.62.56.191:2888:3888:participant;0.0.0.0:2181
server.2=0.0.0.0:2888:3888:participant;0.0.0.0:2181
server.3=10.62.56.193:2888:3888:participant;0.0.0.0:2181

2. But the cluster doesn't really work, I saw the errors:

0:0:2181)(secure=disabled):Learner@273] - Unexpected exception, tries=3, remaining init limit=16997, connecting to /0.0.0.0:2888
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:204)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.apache.zookeeper.server.quorum.Learner.sockConnect(Learner.java:227)
at org.apache.zookeeper.server.quorum.Learner.connectToLeader(Learner.java:256)
at org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:74)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1064)
2015-12-15 04:35:00,403 [myid:1] - INFO
2015-12-15 04:35:00,585 [myid:1] - INFO [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):Follower@198] - shutdown called
java.lang.Exception: shutdown Follower
at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:198)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1068)
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
Important
2 years, 16 weeks ago dynamic reconfiguration 0|i2pwlr:
ZooKeeper ZOOKEEPER-2342

Migrate to Log4J 2.

Bug Open Major Unresolved Chris Nauroth Chris Nauroth Chris Nauroth 09/Dec/15 19:59   25/Feb/20 11:50     3.7.0     3 24   ZOOKEEPER-1371 removed our source code dependency on Log4J. It appears that this also removed the Log4J SLF4J binding jar from the runtime classpath. Without any SLF4J binding jar available on the runtime classpath, it is impossible to write logs.

This JIRA investigated migration to Log4J 2 as a possible path towards resolving the bug introduced by ZOOKEEPER-1371. At this point, we know this is not feasible short-term. This JIRA remains open to track long-term migration to Log4J 2.
9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
3 weeks, 5 days ago 0|i2poyf:
ZooKeeper ZOOKEEPER-2341

bin/zkEnv.cmd needs quotes around %JAVA%

Bug Resolved Trivial Duplicate Unassigned Karl Mortensen Karl Mortensen 09/Dec/15 17:13   09/Dec/15 17:24 09/Dec/15 17:24 3.4.6       0 2   I just downloaded ZooKeeper 3.4.7 (wouldn't let me put that version in the "Affects Version/s" field) and it doesn't work out of the box on Windows 7, which is brutal for folks who don't understand.

It complains that you don't have JAVA_HOME set right if you have it set to a path with spaces e.g. C:\program files\java\blah will fail.

All the following need quotes around and %VARIABLE% expansions to deal with potential spaces in the path:

* bin/zkCli.cmd
* bin/zkEnv.cmd
* bin/zkServer.cmd

Should be a trivial fix.

Definition of Done:
zkCli.cmd, zkEnv.cmd and zkServer.cm work out of the box on Windows 7.
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
4 years, 15 weeks, 1 day ago 0|i2ponj:
ZooKeeper ZOOKEEPER-2340

JMX is disabled even if JMXDISABLE is false

Bug Closed Minor Fixed Mohammad Arshad Neha Bathra Neha Bathra 09/Dec/15 00:33   21/Jul/16 16:18 10/Dec/15 00:33   3.4.8, 3.5.2, 3.6.0     0 7   Currently, to enable jmx for zookeeper, need to comment the property JMXDISABLE as JMXDISABLE=false continues to disable JMX. 9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
4 years, 15 weeks ago 0|i2pn53:
ZooKeeper ZOOKEEPER-2339

Document differences in ZooKeeper usage on Windows.

Improvement Open Major Unresolved Unassigned Chris Nauroth Chris Nauroth 08/Dec/15 18:39   08/Dec/15 18:39       documentation   0 1   The ZooKeeper documentation focuses primarily on usage for Unix-like systems. There are some small differences in usage on Windows. It would be good to enhance the documentation to cover this. 9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
4 years, 15 weeks, 2 days ago 0|i2pmmn:
ZooKeeper ZOOKEEPER-2338

c bindings should create socket's with SOCK_CLOEXEC to avoid fd leaks on fork/exec

Bug Resolved Major Fixed Radu Brumariu James DeFelice James DeFelice 08/Dec/15 11:57   19/Mar/19 20:25 06/Dec/17 17:59 3.5.3, 3.4.11, 3.6.0 3.5.4, 3.6.0 c client   0 5 0 600   I've observed socket FD leaks in Apache Mesos when using ZK to coordinate master leadership: https://issues.apache.org/jira/browse/MESOS-4065 100% 100% 600 0 pull-request-available 9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
2 years, 8 weeks, 6 days ago 0|i2plxb:
ZooKeeper ZOOKEEPER-2337

Fake "invalid" hostnames used in tests are sometimes valid

Bug Closed Major Duplicate Timothy James Ward Timothy James Ward Timothy James Ward 08/Dec/15 10:14   19/Dec/19 18:01 08/Dec/15 10:44 3.4.7, 3.5.1, 3.6.0 3.4.8, 3.5.2     0 2   Some of the ZooKeeper tests use "fake" hostnames to trigger host resolution failures. The problem with this is that it uses valid hostnames which are sometimes configured in VMs.

At the moment I am unable to build cleanly because I get test failures on the two test methods that do this. The tests work equally well if syntactically invalid hostnames are used, and the test cases become more portable at the same time.

The affected test cases are:

org.apache.zookeeper.test.StaticHostProviderTest.testTwoInvalidHostAddresses and org.apache.zookeeper.test.StaticHostProviderTest.testOneInvalidHostAddresses

See GitHub pull request https://github.com/apache/zookeeper/pull/48 for a proposed fix
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
4 years, 15 weeks, 2 days ago 0|i2plrz:
ZooKeeper ZOOKEEPER-2336

Jenkins not working due to old SVN

Test Resolved Major Fixed Akihiro Suda Akihiro Suda Akihiro Suda 07/Dec/15 03:12   03/Mar/16 02:58 03/Mar/16 02:58     build 06/Dec/15 0 3   Jenkins seems not working since Build #2976 (Dec 6, 2015) due to SVN.
https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/2976/console
{panel}
[exec] svn: E155036: Please see the 'svn upgrade' command
[exec] svn: E155036: The working copy at '/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/trunk'
[exec] is too old (format 10) to work with client version '1.8.8 (r1568071)' (expects format 31). You need to upgrade the working copy first.
[exec]
{panel}
9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
4 years, 3 weeks ago 0|i2piof:
ZooKeeper ZOOKEEPER-2335

Java Compilation Error in ClientCnxn.java

Bug Resolved Major Fixed Mohammad Arshad Mohammad Arshad Mohammad Arshad 06/Dec/15 00:28   16/Oct/16 11:23 06/Dec/15 14:01   3.5.2, 3.6.0 java client, server   0 4   There are some compilation error in latest trunk code.
{code}
[javac] D:\gitHome\zookeeperTrunk\src\java\main\org\apache\zookeeper\ClientCnxn.java:49: error: package org.apache.log4j does not exist
[javac] import org.apache.log4j.MDC;
[javac] ^
[javac] D:\gitHome\zookeeperTrunk\src\java\main\org\apache\zookeeper\ClientCnxn.java:1108: error: cannot find symbol
[javac] MDC.put("myid", hostPort);
[javac] ^
[javac] symbol: variable MDC
[javac] location: class ClientCnxn.SendThread
[javac] 2 errors

{code}

This compilation error got introduced by ZOOKEEPER-2330 patch. This patch used log4j api and log4 dependency has already been removed by ZOOKEEPER-1371
9223372036854775807 No Perforce job exists for this issue. 2 9223372036854775807
4 years, 15 weeks, 4 days ago 0|i2pi0n:
ZooKeeper ZOOKEEPER-2334

Zookeeper Archives Out Date

Bug Resolved Major Fixed Flavio Paiva Junqueira Elias Levy Elias Levy 04/Dec/15 15:19   16/Dec/15 10:35 16/Dec/15 10:35         0 5   The Zookeeper download page and mirrors only track the latest version of the mirror release versions. The page has a link to the archives page at archive.apache.org, but that page is missing all releases after 3.3.2. That means there are a large number of releases that disappear from the official download site when a new release is published.

In my particular case I was building a container based on 3.4.6. Once 3.4.7 came out my build broke and it cannot be fixed as 3.4.7 can't be downloaded from anywhere official.
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
4 years, 14 weeks, 1 day ago 0|i2pfev:
ZooKeeper ZOOKEEPER-2332

Zookeeper failed to start for empty txn log

Bug Patch Available Critical Unresolved Shaohui Liu Shaohui Liu Shaohui Liu 27/Nov/15 08:42   14/Dec/19 06:08   3.4.6 3.7.0     2 15   We found that the zookeeper server with version 3.4.6 failed to start for there is a empty txn log in log dir.
I think we should skip the empty log file during restoring the datatree.
Any suggestion?

{code}
2015-11-27 19:16:16,887 [myid:] - ERROR [main:ZooKeeperServerMain@63] - Unexpected exception, exiting abnormally
java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:392)
at org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63)
at org.apache.zookeeper.server.persistence.FileHeader.deserialize(FileHeader.java:64)
at org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.inStreamCreated(FileTxnLog.java:576)
at org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.createInputArchive(FileTxnLog.java:595)
at org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.goToNextLog(FileTxnLog.java:561)
at org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.next(FileTxnLog.java:643)
at org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:158)
at org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:223)
at org.apache.zookeeper.server.ZooKeeperServer.loadData(ZooKeeperServer.java:272)
at org.apache.zookeeper.server.ZooKeeperServer.startdata(ZooKeeperServer.java:399)
at org.apache.zookeeper.server.NIOServerCnxnFactory.startup(NIOServerCnxnFactory.java:122)
at org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:113)
at org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:86)
at org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:52)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:116)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:78)
{code}
9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
1 year, 17 weeks ago 0|i2p0fr:
ZooKeeper ZOOKEEPER-2331

Many instances of ZookeeperSaslClient in one process - every instance should use different jaas config section.

Wish Resolved Minor Duplicate Unassigned Piotr Chmieliński Piotr Chmieliński 26/Nov/15 11:39   24/Mar/16 01:18 28/Nov/15 18:11     java client   0 2   First of all I don't know if "Wish" is the best task type for it. I just want to ask you a question.

Why login field in ZooKeeperSaslClient is a static one?
https://github.com/apache/zookeeper/blob/trunk/src/java/main/org/apache/zookeeper/client/ZooKeeperSaslClient.java#L81

I'd like to have many zookeeper clients in one process. I want each of them to read different section from jaas config. I know that I can specify which one should be read by setting system property:
https://github.com/apache/zookeeper/blob/trunk/src/java/main/org/apache/zookeeper/client/ZooKeeperSaslClient.java#L114

Unfortunately login field is static and it is instantiated during creation of first ZooKeeperSaslClient instance.

Maybe there is some reason behind decision of making "login" static - if yes, could you please explain it?
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
4 years, 16 weeks, 5 days ago 0|i2ozhz:
ZooKeeper ZOOKEEPER-2330

ZooKeeper close API does not close Login thread.

Bug Closed Major Fixed Mohammad Arshad Mohammad Arshad Mohammad Arshad 23/Nov/15 05:03   21/Jul/16 16:18 08/May/16 16:24 3.5.0 3.5.2, 3.6.0 java client   0 5   When kerberos is used as authentication mechanism, one login thread runs in the background for ZooKeeper client as well ZooKeepr server.
This problem is related to Zookeeper client and the scenario is as follows:

# Main application connects to Zookeeper
{code}
ZooKeeper zooKeeper = new ZooKeeper(zookeeperConnectionString, sessionTimeout, this)
{code}
# Completes it is work with zookeeper
# calls close() on zookeeper, and continues with rest of the application specific work

Thread dump, taken after 3rd step, shows that login thread is still alive
{code}
"Thread-1" daemon prio=6 tid=0x04842c00 nid=0x1f04 waiting on condition [0x05b7f000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at org.apache.zookeeper.Login$1.run(Login.java:180)
at java.lang.Thread.run(Thread.java:722)
{code}
9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
3 years, 45 weeks, 4 days ago 0|i2os1j:
ZooKeeper ZOOKEEPER-2329

Clear javac and javadoc warning from zookeeper

Bug Closed Minor Fixed Mohammad Arshad Mohammad Arshad Mohammad Arshad 21/Nov/15 16:41   21/Jul/16 16:18 21/Nov/15 18:35   3.5.2     0 4   Currently ZooKeeper java code has 10 javac and 1 javadoc warning. These should be removed.

*javac warnings*
{noformat}
[javac] Compiling 228 source files to D:\gitHome\zookeeperTrunk\build\classes
[javac] D:\gitHome\zookeeperTrunk\src\java\main\org\apache\zookeeper\ZooKeeperMain.java:226: warning: [rawtypes] found raw type: List
[javac] List args = new LinkedList();
[javac] ^
[javac] missing type arguments for generic class List<E>
[javac] where E is a type-variable:
[javac] E extends Object declared in interface List
[javac] D:\gitHome\zookeeperTrunk\src\java\main\org\apache\zookeeper\ZooKeeperMain.java:226: warning: [rawtypes] found raw type: LinkedList
[javac] List args = new LinkedList();
[javac] ^
[javac] missing type arguments for generic class LinkedList<E>
[javac] where E is a type-variable:
[javac] E extends Object declared in class LinkedList
[javac] D:\gitHome\zookeeperTrunk\src\java\main\org\apache\zookeeper\ZooKeeperMain.java:233: warning: [unchecked] unchecked call to add(E) as a member of the raw type List
[javac] args.add(value);
[javac] ^
[javac] where E is a type-variable:
[javac] E extends Object declared in interface List
[javac] D:\gitHome\zookeeperTrunk\src\java\main\org\apache\zookeeper\ZooKeeperMain.java:239: warning: [unchecked] unchecked conversion
[javac] cmdArgs = args;
[javac] ^
[javac] required: List<String>
[javac] found: List
[javac] D:\gitHome\zookeeperTrunk\src\java\main\org\apache\zookeeper\jmx\ManagedUtil.java:62: warning: [rawtypes] found raw type: Enumeration
[javac] Enumeration enumer = r.getCurrentLoggers();
[javac] ^
[javac] missing type arguments for generic class Enumeration<E>
[javac] where E is a type-variable:
[javac] E extends Object declared in interface Enumeration
[javac] D:\gitHome\zookeeperTrunk\src\java\main\org\apache\zookeeper\server\admin\AdminServer.java:33: warning: [serial] serializable class AdminServerException has no definition of serialVersionUID
[javac] public class AdminServerException extends Exception {
[javac] ^
[javac] D:\gitHome\zookeeperTrunk\src\java\main\org\apache\zookeeper\server\admin\JettyAdminServer.java:142: warning: [serial] serializable class JettyAdminServer.CommandServlet has no definition of serialVersionUID
[javac] private class CommandServlet extends HttpServlet {
[javac] ^
[javac] D:\gitHome\zookeeperTrunk\src\java\main\org\apache\zookeeper\server\util\KerberosUtil.java:39: warning: [rawtypes] found raw type: Class
[javac] getInstanceMethod = classRef.getMethod("getInstance", new Class[0]);
[javac] ^
[javac] missing type arguments for generic class Class<T>
[javac] where T is a type-variable:
[javac] T extends Object declared in class Class
[javac] D:\gitHome\zookeeperTrunk\src\java\main\org\apache\zookeeper\server\util\KerberosUtil.java:42: warning: [rawtypes] found raw type: Class
[javac] new Class[0]);
[javac] ^
[javac] missing type arguments for generic class Class<T>
[javac] where T is a type-variable:
[javac] T extends Object declared in class Class
[javac] D:\gitHome\zookeeperTrunk\src\java\main\org\apache\zookeeper\server\util\OSMXBean.java:89: warning: [rawtypes] found raw type: Class
[javac] new Class[0]);
[javac] ^
[javac] missing type arguments for generic class Class<T>
[javac] where T is a type-variable:
[javac] T extends Object declared in class Class
[javac] 10 warnings
{noformat}
*javadoc warning*
{noformat}
[javadoc] D:\gitHome\zookeeperTrunk\src\java\main\org\apache\zookeeper\server\PurgeTxnLog.java:172: warning - @return tag has no arguments.
{noformat}
9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
4 years, 17 weeks, 4 days ago 0|i2or27:
ZooKeeper ZOOKEEPER-2328

Connection broken for id 1

Bug Resolved Major Not A Problem Unassigned Arun Arun 20/Nov/15 18:48   19/Apr/16 10:18 19/Apr/16 10:18 3.4.6   quorum   0 4   zookeeper-3.4.6 Hi, We have a 3 node zookeeper running with 3.4.6 version. myId 1 (leader) and 2 (follower) are working fine, myId 3 node starts successfully but when we check the status, we see below error, we also do not see this instance taking a load

Any help will be highly appreciated.

$./zkServer.sh status
JMX enabled by default
Using config: ../zookeeper/zookeeper-3.4.6/conf/zoo.cfg
Error contacting service. It is probably not running.

Server Logs:

2015-11-20 23:38:41,863 [myid:3] - WARN [RecvWorker:1:QuorumCnxManager$RecvWorker@780] - Connection broken for id 1, my id = 3, error =
java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:189)
at java.net.SocketInputStream.read(SocketInputStream.java:121)
at java.net.SocketInputStream.read(SocketInputStream.java:203)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:765)
2015-11-20 23:38:41,863 [myid:3] - WARN [RecvWorker:2:QuorumCnxManager$RecvWorker@780] - Connection broken for id 2, my id = 3, error =
java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:189)
at java.net.SocketInputStream.read(SocketInputStream.java:121)
at java.net.SocketInputStream.read(SocketInputStream.java:203)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
2015-11-20 23:23:33,320 [myid:] - INFO [main:QuorumPeerConfig@103] - Reading configuration from: ../zookeeper/zookeeper-3.4.6/conf/zoo.cfg
2015-11-20 23:23:33,344 [myid:] - INFO [main:QuorumPeerConfig@340] - Defaulting to majority quorums
2015-11-20 23:23:33,351 [myid:3] - INFO [main:DatadirCleanupManager@78] - autopurge.snapRetainCount set to 3
2015-11-20 23:23:33,352 [myid:3] - INFO [main:DatadirCleanupManager@79] - autopurge.purgeInterval set to 0
2015-11-20 23:23:33,353 [myid:3] - INFO [main:DatadirCleanupManager@101] - Purge task is not scheduled.
2015-11-20 23:23:33,382 [myid:3] - INFO [main:QuorumPeerMain@127] - Starting quorum peer
2015-11-20 23:23:33,410 [myid:3] - INFO [main:NIOServerCnxnFactory@94] - binding to port 0.0.0.0/0.0.0.0:2181
2015-11-20 23:23:33,452 [myid:3] - INFO [main:QuorumPeer@959] - tickTime set to 2000
2015-11-20 23:23:33,452 [myid:3] - INFO [main:QuorumPeer@979] - minSessionTimeout set to -1
2015-11-20 23:23:33,453 [myid:3] - INFO [main:QuorumPeer@990] - maxSessionTimeout set to -1
2015-11-20 23:23:33,453 [myid:3] - INFO [main:QuorumPeer@1005] - initLimit set to 5
2015-11-20 23:23:33,493 [myid:3] - INFO [Thread-1:QuorumCnxManager$Listener@504] - My election bind port: <host_name>/<IP_address>:3888
2015-11-20 23:23:33,512 [myid:3] - INFO [QuorumPeer[myid=3]/0:0:0:0:0:0:0:0:2181:QuorumPeer@714] - LOOKING
2015-11-20 23:23:33,515 [myid:3] - INFO [QuorumPeer[myid=3]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@815] - New election. My id = 3, proposed zxid=0x0
2015-11-20 23:23:33,528 [myid:3] - INFO [WorkerReceiver[myid=3]:FastLeaderElection@597] - Notification: 1 (message format version), 3 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 3 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2015-11-20 23:23:33,731 [myid:3] - INFO [QuorumPeer[myid=3]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@849] - Notification time out: 400
2015-11-20 23:23:33,732 [myid:3] - INFO [WorkerReceiver[myid=3]:FastLeaderElection@597] - Notification: 1 (message format version), 3 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 3 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2015-11-20 23:23:34,136 [myid:3] - INFO [QuorumPeer[myid=3]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@849] - Notification time out: 800
2015-11-20 23:23:34,137 [myid:3] - INFO [WorkerReceiver[myid=3]:FastLeaderElection@597] - Notification: 1 (message format version), 3 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 3 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2015-11-20 23:23:34,938 [myid:3] - INFO [QuorumPeer[myid=3]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@849] - Notification time out: 1600
2015-11-20 23:23:34,939 [myid:3] - INFO [WorkerReceiver[myid=3]:FastLeaderElection@597] - Notification: 1 (message format version), 3 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 3 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2015-11-20 23:23:36,540 [myid:3] - INFO [QuorumPeer[myid=3]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@849] - Notification time out: 3200
2015-11-20 23:23:36,540 [myid:3] - INFO [WorkerReceiver[myid=3]:FastLeaderElection@597] - Notification: 1 (message format version), 3 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 3 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2015-11-20 23:23:39,741 [myid:3] - INFO [QuorumPeer[myid=3]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@849] - Notification time out: 6400
2015-11-20 23:23:39,742 [myid:3] - INFO [WorkerReceiver[myid=3]:FastLeaderElection@597] - Notification: 1 (message format version), 3 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 3 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 48 weeks, 2 days ago 0|i2oqi7:
ZooKeeper ZOOKEEPER-2327

"ConnectionLoss for /dog"

Bug Open Major Unresolved Unassigned Andrew Pennebaker Andrew Pennebaker 19/Nov/15 20:16   12/Dec/16 03:36   3.5.0 3.3.6, 3.5.1     0 3   Zookeeper v3.5.0-alpha experiences an error when trying to create a simple data node.

Versions before and after 3.5.0-alpha work just fine, but this specific version oddly fails to create data nodes.

Source:

https://github.com/mcandre/docker-zookeeper/tree/3.5.0-alpha

Trace:

```
$ git clone git@github.com:mcandre/docker-zookeeper.git
$ cd docker-zookeeper.git
$ git checkout 3.5.0-alpha
$ make
CONTAINER=$(docker run -d -p 2181:2181 -p 2888:2888 -p 3888:3888 mcandre/docker-zookeeper:3.5.0-alpha)
docker exec $CONTAINER sh -c 'echo "create /dog moon" | zkCli.sh'
Connecting to localhost:2181
2015-11-20 01:05:54,951 [myid:] - INFO [main:Environment@109] - Client environment:zookeeper.version=3.5.0-alpha-1615249, built on 08/01/2014 22:13 GMT
2015-11-20 01:05:54,958 [myid:] - INFO [main:Environment@109] - Client environment:host.name=190002648bbc
2015-11-20 01:05:54,958 [myid:] - INFO [main:Environment@109] - Client environment:java.version=1.7.0_85
2015-11-20 01:05:54,969 [myid:] - INFO [main:Environment@109] - Client environment:java.vendor=Oracle Corporation
2015-11-20 01:05:54,970 [myid:] - INFO [main:Environment@109] - Client environment:java.home=/usr/lib/jvm/java-7-openjdk-amd64/jre
2015-11-20 01:05:54,970 [myid:] - INFO [main:Environment@109] - Client environment:java.class.path=/zookeeper-3.5.0-alpha/bin/../build/classes:/zookeeper-3.5.0-alpha/bin/../build/lib/*.jar:/zookeeper-3.5.0-alpha/bin/../lib/slf4j-log4j12-1.7.5.jar:/zookeeper-3.5.0-alpha/bin/../lib/slf4j-api-1.7.5.jar:/zookeeper-3.5.0-alpha/bin/../lib/servlet-api-2.5-20081211.jar:/zookeeper-3.5.0-alpha/bin/../lib/netty-3.7.0.Final.jar:/zookeeper-3.5.0-alpha/bin/../lib/log4j-1.2.16.jar:/zookeeper-3.5.0-alpha/bin/../lib/jline-2.11.jar:/zookeeper-3.5.0-alpha/bin/../lib/jetty-util-6.1.26.jar:/zookeeper-3.5.0-alpha/bin/../lib/jetty-6.1.26.jar:/zookeeper-3.5.0-alpha/bin/../lib/javacc.jar:/zookeeper-3.5.0-alpha/bin/../lib/jackson-mapper-asl-1.9.11.jar:/zookeeper-3.5.0-alpha/bin/../lib/jackson-core-asl-1.9.11.jar:/zookeeper-3.5.0-alpha/bin/../lib/commons-cli-1.2.jar:/zookeeper-3.5.0-alpha/bin/../zookeeper-3.5.0-alpha.jar:/zookeeper-3.5.0-alpha/bin/../src/java/lib/*.jar:/zookeeper-3.5.0-alpha/bin/../conf:
2015-11-20 01:05:54,971 [myid:] - INFO [main:Environment@109] - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib
2015-11-20 01:05:54,972 [myid:] - INFO [main:Environment@109] - Client environment:java.io.tmpdir=/tmp
2015-11-20 01:05:54,972 [myid:] - INFO [main:Environment@109] - Client environment:java.compiler=<NA>
2015-11-20 01:05:54,973 [myid:] - INFO [main:Environment@109] - Client environment:os.name=Linux
2015-11-20 01:05:54,973 [myid:] - INFO [main:Environment@109] - Client environment:os.arch=amd64
2015-11-20 01:05:54,974 [myid:] - INFO [main:Environment@109] - Client environment:os.version=4.0.9-boot2docker
2015-11-20 01:05:54,974 [myid:] - INFO [main:Environment@109] - Client environment:user.name=root
2015-11-20 01:05:54,974 [myid:] - INFO [main:Environment@109] - Client environment:user.home=/root
2015-11-20 01:05:54,975 [myid:] - INFO [main:Environment@109] - Client environment:user.dir=/
2015-11-20 01:05:54,975 [myid:] - INFO [main:Environment@109] - Client environment:os.memory.free=26MB
2015-11-20 01:05:54,977 [myid:] - INFO [main:Environment@109] - Client environment:os.memory.max=247MB
2015-11-20 01:05:54,977 [myid:] - INFO [main:Environment@109] - Client environment:os.memory.total=30MB
2015-11-20 01:05:54,987 [myid:] - INFO [main:ZooKeeper@709] - Initiating client connection, connectString=localhost:2181 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@3c407d5
Welcome to ZooKeeper!
2015-11-20 01:05:55,062 [myid:] - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread@1093] - Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
2015-11-20 01:05:55,099 [myid:] - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread@963] - Socket connection established to localhost/127.0.0.1:2181, initiating session
JLine support is enabled
2015-11-20 01:05:55,154 [myid:] - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread@1209] - Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect
[zk: localhost:2181(CONNECTING) 0] create /dog moon
Exception in thread "main" org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /dog
at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:1067)
at org.apache.zookeeper.cli.CreateCommand.exec(CreateCommand.java:78)
at org.apache.zookeeper.ZooKeeperMain.processZKCmd(ZooKeeperMain.java:670)
at org.apache.zookeeper.ZooKeeperMain.processCmd(ZooKeeperMain.java:573)
at org.apache.zookeeper.ZooKeeperMain.executeLine(ZooKeeperMain.java:356)
at org.apache.zookeeper.ZooKeeperMain.run(ZooKeeperMain.java:316)
at org.apache.zookeeper.ZooKeeperMain.main(ZooKeeperMain.java:276)
make: *** [run] Error 1
```
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 51 weeks, 6 days ago 0|i2oofr:
ZooKeeper ZOOKEEPER-2326

Include connected server address:port in log

Improvement Closed Minor Fixed Mohammad Arshad Mohammad Arshad Mohammad Arshad 18/Nov/15 17:23   21/Jul/16 16:18 05/Dec/15 16:46   3.5.2, 3.6.0 java client   0 3   Currently ZooKeeper client log contains blank myid as bellow
{noformat}
2015-11-18 23:46:39,045 [myid:] - INFO [main-SendThread(192.168.1.3:2183):ClientCnxn$SendThread@1138] - Opening socket connection to server 192.168.1.3/192.168.1.3:2183. Will attempt to SASL-authenticate using Login Context section 'Client'
2015-11-18 23:46:40,387 [myid:] - WARN [main-SendThread(192.168.1.3:2183):ClientCnxn$SendThread@1206] - Client session timed out, have not heard from server in 1499ms for sessionid 0x200009eb6510002
2015-11-18 23:46:40,387 [myid:] - INFO [main-SendThread(192.168.1.3:2183):ClientCnxn$SendThread@1254] - Client session timed out, have not heard from server in 1499ms for sessionid 0x200009eb6510002, closing socket connection and attempting reconnect
2015-11-18 23:46:41,323 [myid:] - INFO [main-SendThread(192.168.1.3:2181):ZooKeeperSaslClient@235] - Client will use DIGEST-MD5 as SASL mechanism.
{noformat}

myid is blank. it is fine as at client side myid(serverid ) does not make any sense. But we can assign myid serverIP: port which will be a very helpful information in analysing the issues. So we after fix we can have log as bellow
{noformat}
2015-11-19 03:51:27,254 [myid:192.168.1.3:2183] - INFO [main-SendThread(192.168.1.3:2183):Login@290] - successfully logged in.
2015-11-19 03:51:27,270 [myid:192.168.1.3:2183] - INFO [Thread-0:Login$1@124] - TGT refresh thread started.
2015-11-19 03:51:27,270 [myid:192.168.1.3:2183] - INFO [main-SendThread(192.168.1.3:2183):ZooKeeperSaslClient$1@297] - Client will use GSSAPI as SASL mechanism.
2015-11-19 03:51:27,270 [myid:192.168.1.3:2183] - INFO [Thread-0:Login@298] - TGT valid starting at: Thu Nov 19 03:51:27 IST 2015
2015-11-19 03:51:27,270 [myid:192.168.1.3:2183] - INFO [Thread-0:Login@299] - TGT expires: Thu Nov 19 03:53:27 IST 2015
2015-11-19 03:51:27,270 [myid:192.168.1.3:2183] - INFO [Thread-0:Login$1@178] - TGT refresh sleeping until: Thu Nov 19 03:53:05 IST 2015
2015-11-19 03:51:27,285 [myid:192.168.1.3:2183] - INFO [main-SendThread(192.168.1.3:2183):ClientCnxn$SendThread@1141] - Opening socket connection to server 192.168.1.3/192.168.1.3:2183. Will attempt to SASL-authenticate using Login Context section 'Client'
2015-11-19 03:51:27,301 [myid:192.168.1.3:2183] - INFO [main-SendThread(192.168.1.3:2183):ClientCnxn$SendThread@981] - Socket connection established, initiating session, client: /192.168.1.2:53117, server: 192.168.1.3/192.168.1.3:2183
2015-11-19 03:51:28,632 [myid:192.168.1.3:2183] - WARN [main-SendThread(192.168.1.3:2183):ClientCnxn$SendThread@1209] - Client session timed out, have not heard from server in 1333ms for sessionid 0x0
2015-11-19 03:51:28,632 [myid:192.168.1.3:2183] - INFO [main-SendThread(192.168.1.3:2183):ClientCnxn$SendThread@1257] - Client session timed out, have not heard from server in 1333ms for sessionid 0x0, closing socket connection and attempting reconnect
2015-11-19 03:51:29,147 [myid:192.168.1.3:2181] - INFO [main-SendThread(192.168.1.3:2181):ZooKeeperSaslClient$1@297] - Client will use GSSAPI as SASL mechanism.
2015-11-19 03:51:29,152 [myid:192.168.1.3:2181] - INFO [main-SendThread(192.168.1.3:2181):ClientCnxn$SendThread@1141] - Opening socket connection to server 192.168.1.3/192.168.1.3:2181. Will attempt to SASL-authenticate using Login Context section 'Client'
2015-11-19 03:51:29,154 [myid:192.168.1.3:2181] - INFO [main-SendThread(192.168.1.3:2181):ClientCnxn$SendThread@981] - Socket connection established, initiating session, client: /192.168.1.2:53118, server: 192.168.1.3/192.168.1.3:2181
2015-11-19 03:51:30,487 [myid:192.168.1.3:2181] - WARN [main-SendThread(192.168.1.3:2181):ClientCnxn$SendThread@1209] - Client session timed out, have not heard from server in 1333ms for sessionid 0x0
{noformat}
9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
4 years, 15 weeks, 5 days ago 0|i2oluv:
ZooKeeper ZOOKEEPER-2325

Data inconsistency if all snapshots empty or missing

Bug Resolved Critical Fixed Andrew Grasso Andrew Grasso Andrew Grasso 18/Nov/15 17:21   21/Aug/19 12:59 26/Oct/18 06:04 3.4.6 3.5.4, 3.6.0 server   1 15 3600 3000 600 16% When loading state from snapshots on startup, FileTxnSnapLog.java ignores the result of FileSnap.deserialize, which is -1L if no valid snapshots are found. Recovery proceeds with dt.lastProcessed == 0, its initial value.

The result is that Zookeeper will process the transaction logs and then begin serving requests with a different state than the rest of the ensemble.

To reproduce:
In a healthy zookeeper cluster of size >= 3, shut down one node.
Either delete all snapshots for this node or change all to be empty files.
Restart the node.

We believe this can happen organically if a node runs out of disk space.
16% 16% 600 3000 3600 pull-request-available 9223372036854775807 No Perforce job exists for this issue. 3 9223372036854775807
1 year, 20 weeks, 6 days ago 0|i2oluf:
ZooKeeper ZOOKEEPER-2324

Make Client authentication mechanism change optional

Improvement Open Major Unresolved Mohammad Arshad Mohammad Arshad Mohammad Arshad 18/Nov/15 16:56   05/Feb/20 07:16     3.7.0, 3.5.8 java client   0 1   Currently if ZooKeeper Client fails to authenticate using kerberos GSSAPI mechanism it automatically switches to DIGEST-MD5 authentication mechanism. This should be configurable. whether to switch to DIGEST-MD5 or fail the authentication and throw {{LoginException}} 9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
4 years, 18 weeks, 1 day ago 0|i2oltr:
ZooKeeper ZOOKEEPER-2323

ZooKeeper client enters into infinite AuthFailedException cycle if its unable to recreate Kerberos ticket

Bug Resolved Major Fixed Mohammad Arshad Mohammad Arshad Mohammad Arshad 18/Nov/15 15:29   11/Aug/16 16:00 11/Aug/16 16:00 3.4.7, 3.5.1 3.5.2 java client   1 5   ZooKeeper client enters into infinite AuthFailedException cycle. For every operation its throws AuthFailedException
Here is the create operation exception
{code}
org.apache.zookeeper.KeeperException$AuthFailedException: KeeperErrorCode = AuthFailed for /continuousRunningZKClient
at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1753)
{code}

This can be reproduced easily with the following steps:

# Reduce the ZooKeeper client principal max life for example set 2 min. use command {color:blue} modprinc -maxlife 2min zkcli {color} in kadmin. (This is done to reduce the issue reproduce time)
# Connect Client to ZooKeeper quorum,let it gets connected and some operations are done successfully
# Disconnect the Client's network, by pulling out the Ethernet cable or by any way. Now the Client is in disconnected state, no operation is expected,Client tries to reconnect to different-different servers in the ZooKeeper quorum.
# After two minutes Client tries to get new Keberos ticket and it fails.
# Connect the Client to network. Client comes in connected state but AuthFailedException for every operation.
9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
3 years, 32 weeks ago 0|i2olnj:
ZooKeeper ZOOKEEPER-2322

Zookeeper server throws NullPointerExceptions while revalidating the session to the client

Bug Open Minor Unresolved Unassigned Neha Bathra Neha Bathra 16/Nov/15 06:38   16/Nov/15 06:38           0 2   3 nodes Suse 11 SP3 cluster This happened with a long run scenario where the client is connected to Zk and leader re-election happens in some time interval
StackTrace:

2015-11-05 03:15:42,953 [myid:1] - WARN [NIOWorkerThread-6:WorkerService$ScheduledWorkRequest@164] - Unexpected exception
java.lang.NullPointerException
at org.apache.zookeeper.server.quorum.LearnerZooKeeperServer.revalidateSession(LearnerZooKeeperServer.java:93)
at org.apache.zookeeper.server.ZooKeeperServer.reopenSession(ZooKeeperServer.java:692)
at org.apache.zookeeper.server.ZooKeeperServer.processConnectRequest(ZooKeeperServer.java:1039)
at org.apache.zookeeper.server.NIOServerCnxn.readConnectRequest(NIOServerCnxn.java:434)
at org.apache.zookeeper.server.NIOServerCnxn.readPayload(NIOServerCnxn.java:180)
at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:340)
at org.apache.zookeeper.server.NIOServerCnxnFactory$IOWorkRequest.doWork(NIOServerCnxnFactory.java:536)
at org.apache.zookeeper.server.WorkerService$ScheduledWorkRequest.run(WorkerService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
4 years, 18 weeks, 3 days ago 0|i2ofqn:
ZooKeeper ZOOKEEPER-2321

C-client session watcher removal is not thread safe

Bug Open Major Unresolved Unassigned Hadriel Kaplan Hadriel Kaplan 13/Nov/15 19:12   13/Nov/15 19:12   3.5.1   c client   0 1   Invoking the C-client API function {{zoo_set_watcher()}} to remove/change a session event watcher is not a thread-safe operation. The IO thread accesses the session watcher (the one stored in the zhandle_t.watcher member) and copies its value into completion events, which are then later processed by the completion thread. This happens when it's processing session events, such as session connected/connecting/expired events.

Meanwhile after the value has been copied by the IO thread, but before the completion thread has used it, the main thread could change the watcher to NULL using {{zoo_set_watcher()}} because the calling application may be free'ing it. The call to {{zoo_set_watcher()}} will return even though the IO and completion threads still have the old watcher pointer value, and the main application cannot safely free it. But since the function call returns, the main application thinks it can free it, and boom goes the dynamite.

So... either there needs to be a lockout while the IO/completion threads process session events, or the {{zoo_set_watcher()}} needs to become asynchronous itself by going through the same processing pipeline to the completion thread and having a completion callback to tell the calling application when it succeeded/failed.
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
4 years, 18 weeks, 6 days ago 0|i2oe6f:
ZooKeeper ZOOKEEPER-2320

C-client crashes when removing watcher asynchronously in "local" mode

Bug Patch Available Major Unresolved Abraham Fine Hadriel Kaplan Hadriel Kaplan 12/Nov/15 18:37   16/Dec/18 10:01   3.5.1   c client   0 5 0 4200   The C-client library will crash when invoking the asynchronous {{zoo_aremove_watchers()}} API function with the '{{local}}' argument set to 1.

The reason is: if the local argument is 1/true, then the code does '{{notify_sync_completion((struct sync_completion *)data);}}' But casting the '{{data}}' variable to a {{sync_completion}} struct pointer is bogus/invalid, and when it's later handles as that struct pointer it's accessing invalid memory.

As a side note: it will work ok when called _synchronously_ through {{zoo_remove_watchers()}}, because that function creates a {{sync_completion}} struct and passes it to the asynch {{zoo_aremove_watchers()}}, but it will not work ok when the asynch function is used directly for the reason stated perviously.

Another side note: the docs state that setting the 'local' flag makes the C-client remove the watcher "even if there is no server connection" - but really it makes the C-client remove the watcher without notifying the server at *all*, even if the connection to a server is up. (well... that's what it would do if it didn't just crash instead ;)
100% 100% 4200 0 pull-request-available, remove_watches 9223372036854775807 No Perforce job exists for this issue. 2 9223372036854775807
1 year, 17 weeks ago 0|i2obxb:
ZooKeeper ZOOKEEPER-2319

UnresolvedAddressException cause the QuorumCnxManager.Listener exit

Bug Resolved Major Fixed Michael Han Zhaohui Yu Zhaohui Yu 09/Nov/15 18:36   05/Jul/18 07:18 18/Jun/18 00:05 3.4.6 3.4.7, 3.5.0, 3.6.0     0 7 0 1200   Given three nodes, the leader on 2, but some issue with this machine, so I shutdown this machine, and change the host name to another machine.
Then I start the node in the new machine, but the new node can not join.
I found the the 1 and 3's Listener thread exit.

With the code of Listener's run method:
I think we should catch UnresolvedAddressException to avoid the Listener exit.
{noformat}
@Override
public void run() {

while((!shutdown) && (numRetries < 3)){
try {
// bind and accept
receiveConnection(client);

} catch (IOException e) {

}
}
//
}
{noformat}
100% 100% 1200 0 pull-request-available 9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
1 year, 37 weeks ago 0|i2o63b:
ZooKeeper ZOOKEEPER-2318

segfault in auth_completion_func

Bug Resolved Major Duplicate Unassigned Marshall McMullen Marshall McMullen 09/Nov/15 12:26   27/May/16 01:27 27/May/16 01:27 3.5.0   c client   1 5   We have seen some sporadic issues with unexplained segfaults inside auth_completion_func. The interesting thing is we are not using any auth mechanism at all. This happened against this version of the code:

svn.apache.org/repos/asf/zookeeper/trunk@1547702

Here's the stacktrace we are seeing:

{code}
Thread 1 (Thread 0x7f21d13ff700 ? (LWP 5230)):
#0 0x00007f21efff42f0 in auth_completion_func (rc=0, zh=0x7f21e7470800) at src/zookeeper.c:1696
#1 0x00007f21efff7898 in zookeeper_process (zh=0x7f21e7470800, events=2) at src/zookeeper.c:2708
#2 0x00007f21f0006583 in do_io (v=0x7f21e7470800) at src/mt_adaptor.c:440
#3 0x00007f21eeab7e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#4 0x00007f21ed1803fd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#5 0x0000000000000000 in ?? ()
{code}

The offending line in our case is:

1696 LOG_INFO(LOGCALLBACK(zh), "Authentication scheme %s succeeded", zh->auth_h.auth->scheme);

It must be the case that zh->auth_h.auth is NULL for this to happen since the code path returns if zh is NULL.

Interesting log messages around this time:

{code}
Socket [10.170.243.7:2181] zk retcode=-2, errno=115(Operation now in progress): unexpected server response: expected 0xfffffff9, but received 0xfffffff8
Priming connection to [10.170.243.4:2181]: last_zxid=0x370eb4d
initiated connection to server [10.170.243.4:2181]
Oct 13 12:03:21.273384 zookeeper - INFO [NIOServerCxnFactory.AcceptThread:/10.170.243.4:2181:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /10.170.243.4:48523
Oct 13 12:03:21.274321 zookeeper - WARN [NIOWorkerThread-24:ZooKeeperServer@822] - Connection request from old client /10.170.243.4:48523; will be dropped if server is in r-o mode
Oct 13 12:03:21.274452 zookeeper - INFO [NIOWorkerThread-24:ZooKeeperServer@869] - Client attempting to renew session 0x3000011596d004a at /10.170.243.4:48523; client last zxid is 0x30370eb4d; server last zxid is 0x30370eb4d
Oct 13 12:03:21.274584 zookeeper - INFO [NIOWorkerThread-24:Learner@115] - Revalidating client: 0x3000011596d004a
session establishment complete on server [10.170.243.4:2181], sessionId=0x3000011596d004a, negotiated timeout=20000
Oct 13 12:03:21.275693 zookeeper - INFO [QuorumPeer[myid=1]/10.170.243.4:2181:ZooKeeperServer@611] - Established session 0x3000011596d004a with negotiated timeout 20000 for client /10.170.243.4:48523
Oct 13 12:03:24.229590 zookeeper - WARN [NIOWorkerThread-8:NIOServerCnxn@361] - Unable to read additional data from client sessionid 0x3000011596d004a, likely client has closed socket
Oct 13 12:03:24.230018 zookeeper - INFO [NIOWorkerThread-8:NIOServerCnxn@999] - Closed socket connection for client /10.170.243.4:48523 which had sessionid 0x3000011596d004a
Oct 13 12:03:24.230257 zookeeper - WARN [NIOWorkerThread-19:NIOServerCnxn@361] - Unable to read additional data from client sessionid 0x100002743aa0001, likely client has closed socket
{code}
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 42 weeks, 6 days ago 0|i2o5ef:
ZooKeeper ZOOKEEPER-2317

Non-OSGi compatible version

Bug Closed Blocker Fixed Sachin Markus Tippmann Markus Tippmann 09/Nov/15 05:59   20/May/19 13:50 29/May/18 20:30 3.5.1 3.6.0, 3.5.5 build   0 9 0 1800   Karaf OSGi container Bundle cannot be deployed to OSGi container.
Manifest version is not OSGi compatible.

Instead of using 3.5.1-alpha, manifest needs to contain 3.5.1.alpha
100% 100% 1800 0 pull-request-available 9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
1 year, 42 weeks, 1 day ago 0|i2o4tr:
ZooKeeper ZOOKEEPER-2316

comment does not match code logic

Bug Resolved Trivial Fixed Umesh Panchaksharaiah sunhaitao sunhaitao 06/Nov/15 09:34   12/Jul/17 23:07 27/Apr/17 17:21 3.5.1 3.5.4, 3.6.0 server   0 7   when i read the code below, the comment is put in an incorrect,place.
" // in order to be committed, a proposal must be accepted by a quorum "
should be place on top of :
if (!p.hasAllQuorums()) {
return false;
}

---------------------------------------------------------------------------------------
3.5.1 Leader code
// getting a quorum from all necessary configurations
if (!p.hasAllQuorums()) {
return false;
}

// commit proposals in order
if (zxid != lastCommitted+1) {
LOG.warn("Commiting zxid 0x" + Long.toHexString(zxid)
+ " from " + followerAddr + " not first!");
LOG.warn("First is "
+ (lastCommitted+1));
}

// in order to be committed, a proposal must be accepted by a quorum

outstandingProposals.remove(zxid);
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
2 years, 47 weeks ago 0|i2o293:
ZooKeeper ZOOKEEPER-2315

Change client connect zk service timeout log level from Info to Warn level

Improvement Closed Minor Fixed Yiqun Lin Yiqun Lin Yiqun Lin 05/Nov/15 22:29   21/Jul/16 16:18 08/Nov/15 18:13 3.4.6 3.4.7, 3.5.2, 3.6.0 java client   0 4   Recently my the resourmanager of my hadoop cluster is fail suddenly,so I look into the rsourcemanager log.But the log is not helpful for me to direct find the reson until I found the zk timeout info log record.
{code}
2015-11-06 06:34:11,257 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerNode: Assigned container container_1446016482901_292094_01_000140 of capacity <memory:1024, vCores:1> on host mofa2089:41361, which has 30 containers, <memory:31744, vCores:30> used and <memory:9216, vCores:10> available after allocation
2015-11-06 06:34:11,266 INFO org.apache.zookeeper.ClientCnxn: Unable to reconnect to ZooKeeper service, session 0x24f4fd5118e5c6e has expired, closing socket connection
2015-11-06 06:34:11,271 INFO org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: container_1446016482901_292094_01_000105 Container Transitioned from RUNNING to COMPLETED
2015-11-06 06:34:11,271 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSAppAttempt: Completed container: container_1446016482901_292094_01_000105 in state: COMPLETED event:FINISHED
2015-11-06 06:34:11,271 INFO org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=dongwei OPERATION=AM Released Container TARGET=SchedulerApp RESULT=SUCCESS APPID=application_1446016482901_292094 CONTAINERID=container_1446016482901_292094_01_000105
2015-11-06 06:34:11,271 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerNode: Released container container_1446016482901_292094_01_000105 of capacity <memory:1024, vCores:1> on host mofa010079:50991, which currently has 29 containers, <memory:33792, vCores:29> used and <memory:7168, vCores:11> available, release resources=true
2015-11-06 06:34:11,271 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler: Application attempt appattempt_1446016482901_292094_000001 released container container_1446016482901_292094_01_000105 on node: host: mofa010079:50991 #containers=29 available=<memory:7168, vCores:11> used=<memory:33792, vCores:29> with event: FINISHED
2015-11-06 06:34:11,272 INFO org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: container_1446016482901_292094_01_000141 Container Transitioned from NEW to ALLOCATED
2015-11-06 06:34:11,272 INFO org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=dongwei OPERATION=AM Allocated Container TARGET=SchedulerApp RESULT=SUCCESS APPID=application_1446016482901_292094 CONTAINERID=container_1446016482901_292094_01_000141
2015-11-06 06:34:11,272 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerNode: Assigned container container_1446016482901_292094_01_000141 of capacity <memory:1024, vCores:1> on host mofa010079:50991, which has 30 containers, <memory:34816, vCores:30> used and <memory:6144, vCores:10> available after allocation
2015-11-06 06:34:11,295 WARN org.apache.hadoop.yarn.server.resourcemanager.amlauncher.ApplicationMasterLauncher: org.apache.hadoop.yarn.server.resourcemanager.amlauncher.ApplicationMasterLauncher$LauncherThread interrupted. Returning.
2015-11-06 06:34:11,296 INFO org.apache.hadoop.ipc.Server: Stopping server on 8032
2015-11-06 06:34:11,297 INFO org.apache.hadoop.ipc.Server: Stopping IPC Server Responder
2015-11-06 06:34:11,297 INFO org.apache.hadoop.ipc.Server: Stopping server on 8030
2015-11-06 06:34:11,297 INFO org.apache.hadoop.ipc.Server: Stopping IPC Server listener on 8032
2015-11-06 06:34:11,298 INFO org.apache.hadoop.ipc.Server: Stopping IPC Server Responder
2015-11-06 06:34:11,298 INFO org.apache.hadoop.ipc.Server: Stopping server on 8031
2015-11-06 06:34:11,298 INFO org.apache.hadoop.ipc.Server: Stopping IPC Server listener on 8030
2015-11-06 06:34:11,300 INFO org.apache.hadoop.ipc.Server: Stopping IPC Server listener on 80312015-11-06 06:34:11,300 INFO org.apache.hadoop.ipc.Server: Stopping IPC Server Responder
{code}
The problem is solved,but it's too difficult to find the connect zk service time out info from so many info log records.And we will easily to ignore these records.So we should chang these zk seesion timeout log level form info level to warn.
9223372036854775807 No Perforce job exists for this issue. 2 9223372036854775807
4 years, 19 weeks, 3 days ago 0|i2o1fj:
ZooKeeper ZOOKEEPER-2314

Improvements to SASL

Improvement Open Major Unresolved Flavio Paiva Junqueira Flavio Paiva Junqueira Flavio Paiva Junqueira 05/Nov/15 11:26   05/Feb/20 07:15   3.4.6, 3.5.1 3.7.0, 3.5.8 documentation   0 9   ZOOKEEPER-2396, ZOOKEEPER-2397 Points that occur to me right now:

# The login object in ZooKeeperSaslClient is static, which means that if you try to create another client for tests, the login object will be the first one you've set for all runs. I've experienced this with 3.4.6.
# There are a number of properties spread across the code that do not appear in the docs. For example, zookeeper.allowSaslFailedClients isn't documented afaict.
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 2 weeks, 3 days ago 0|i2o0fr:
ZooKeeper ZOOKEEPER-2313

Refactor ZooKeeperServerBean and its subclasses (LeaderBean, ObserverBean, FollowerBean)

Improvement Open Minor Unresolved Rabi Kumar K C Edward Ribeiro Edward Ribeiro 04/Nov/15 05:23   05/Feb/20 07:15   3.4.6 3.7.0, 3.5.8 server   0 2 0 4200   Following the work on ZOOKEEPER-2142, the goal of this ticket is to add a new constructor to ZooKeeperServerBean to pass the name as the parameter, so classes that extend from ZooKeeperServerBean shouldn't need to implement getName() method. Also, ObserverBean seems to be in the wrong package, it should be under server.quorum rather than just server. 100% 100% 4200 0 pull-request-available 9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
4 years, 20 weeks, 1 day ago 0|i2nxlr:
ZooKeeper ZOOKEEPER-2312

Fix arrow direction in the 2-phase commit diagram in Zookeeper internal docs

Bug Open Minor Unresolved Raju Bairishetti Raju Bairishetti Raju Bairishetti 03/Nov/15 22:53   03/Nov/15 23:28       documentation   0 2   https://zookeeper.apache.org/doc/r3.3.3/zookeeperInternals.html

Leader issues *commit request* to followers once the ack received from the followers. But the 2-phase commit diagram shows the direction of commit from Follower to Leader.

[2-phase-commit-image|https://github.com/apache/zookeeper/blob/trunk/src/docs/src/documentation/resources/images/2pc.jpg]
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
4 years, 20 weeks, 1 day ago 0|i2nx67:
ZooKeeper ZOOKEEPER-2311

assert in setup_random

Bug Closed Major Fixed Marshall McMullen Marshall McMullen Marshall McMullen 02/Nov/15 15:24   21/Jul/16 16:18 06/Dec/15 14:34 3.4.7, 3.5.1 3.4.8, 3.5.2, 3.6.0 c client   0 4   We've started seeing an assert failing inside setup_random at line 537:

{code}
528 static void setup_random()
529 {
530 #ifndef _WIN32 // TODO: better seed
531 int seed;
532 int fd = open("/dev/urandom", O_RDONLY);
533 if (fd == -1) {
534 seed = getpid();
535 } else {
536 int rc = read(fd, &seed, sizeof(seed));
537 assert(rc == sizeof(seed));
538 close(fd);
539 }
540 srandom(seed);
541 srand48(seed);
542 #endif
{code}

The core files show:

Program terminated with signal 6, Aborted.
#0 0x00007f9ff665a0d5 in raise () from /lib/x86_64-linux-gnu/libc.so.6
#0 0x00007f9ff665a0d5 in raise () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f9ff665d83b in abort () from /lib/x86_64-linux-gnu/libc.so.6
#2 0x00007f9ff6652d9e in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#3 0x00007f9ff6652e42 in __assert_fail () from /lib/x86_64-linux-gnu/libc.so.6
#4 0x00007f9ff8e4070a in setup_random () at src/zookeeper.c:476
#5 0x00007f9ff8e40d76 in resolve_hosts (zh=0x7f9fe14de400, hosts_in=0x7f9fd700f400 "10.26.200.6:2181,10.26.200.7:2181,10.26.200.8:2181", avec=0x7f9fd87fab60) at src/zookeeper.c:730
#6 0x00007f9ff8e40e87 in update_addrs (zh=0x7f9fe14de400) at src/zookeeper.c:801
#7 0x00007f9ff8e44176 in zookeeper_interest (zh=0x7f9fe14de400, fd=0x7f9fd87fac4c, interest=0x7f9fd87fac50, tv=0x7f9fd87fac80) at src/zookeeper.c:1980
#8 0x00007f9ff8e553f5 in do_io (v=0x7f9fe14de400) at src/mt_adaptor.c:379
#9 0x00007f9ff804de9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007f9ff671738d in clone () from /lib/x86_64-linux-gnu/libc.so.6
#11 0x0000000000000000 in ?? ()

I'm not sure what the underlying cause of this is... But POSIX always allows for a short read(2), and any program MUST check for short reads...

Has anyone else encountered this issue? We are seeing it rather frequently which is concerning.
9223372036854775807 No Perforce job exists for this issue. 2 9223372036854775807
4 years, 15 weeks, 4 days ago 0|i2nua7:
ZooKeeper ZOOKEEPER-2310

Snapshot files must be synced to prevent inconsistency or data loss

Bug Patch Available Major Unresolved Abhishek Rai Abhishek Rai Abhishek Rai 01/Nov/15 01:26   11/Sep/16 04:19   3.4.6   server   0 6   Today, Zookeeper server syncs transaction log files to disk by default, but does not sync snapshot files. Consequently, an untimely crash may result in a lost or incomplete snapshot file. During recovery, if the server finds a valid older snapshot file, it will load it and replay subsequent log(s), skipping the incomplete snapshot file. It's possible that the skipped file had some transactions which are not present in the replayed transaction logs. Since quorum synchronization is based on last transaction ID of each server, this will never get noticed, resulting in inconsistency between servers and possible data loss.

Following sequence of events describes a sample scenario where this can happen:

# Server F is a follower in a Zookeeper ensemble.
# F's most recent valid snapshot file is named "snapshot.10" containing state up to zxid = 10. F is currently writing to the transaction log file "log.11", with the most recent zxid = 20.
# Fresh round of election.
# F receives a few new transactions 21 to 30 from new leader L as the "diff". Current server behavior is to dump current state plus diff to a new snapshot file, "snapshot.30".
# F finalizes the snapshot file, but file contents are still buffered in OS caches. Zookeeper does not sync snapshot file contents to disk.
# F receives a new transaction 31 from the leader, which it appends to the existing transaction log file, "log.11" and syncs the file to disk.
# Server machine crashes or is cold rebooted.
# After recovery, snapshot file "snapshot.30" may not exist or may be empty. See below for why that may happen.
# In either case, F looks for the last finalized snapshot file, finds and loads "snapshot.10". It then replays transactions from "log.11". Ultimately, its last seen zxid will be 31, but it would not have replayed transactions 21 to 30 received via the "diff" from the leader.
# Clients which are connected to F may see different data than clients connected to other members of the ensemble, violating single system image invariant. Also, if F were to become a leader at some point, it could use its state to seed other servers, and they all could lose the writes in the missing interval above.

*Notes:*
- Reason why snapshot file may be missing or incomplete:
-- Zookeeper does not sync the data directory after creating a snapshot file. Even if a newly created file is synced to disk, if the corresponding directory entry is not, then the file will not be visible in the namespace.
-- Zookeeper does not sync snapshot files. So, they may be empty or incomplete during recovery from an untimely crash.
- In step (6) above, the server could also have written the new transaction 31 to a new log file, "log.31". The final outcome would still be the same.

We are able to deterministically reproduce this problem using the following steps:

# Create a new Zookeeper ensemble on 3 hosts: A, B, and C.
# Ensured each server has at least one snapshot file in its data dir.
# Stop Zookeeper process on server A.
# Slow down disk syncs on server A (see example script below). This ensures that snapshot files written by Zookeeper don't make it to disk spontaneously. Log files will be written to disk as Zookeeper explicitly issues a sync call on such files.
# Connect to server B and create a new znode /test1.
# Start Zookeeper process on A, wait for it to write a new snapshot to its datadir. This snapshot would contain /test1 but it won’t be synced to disk yet.
# Connect to A and verify that /test1 is visible.
# Connect to B and create another znode /test2. This will cause A’s transaction log to grow further to receive /test2.
# Cold reboot A.
# A’s last snapshot is a zero-sized file or is missing altogether since it did not get synced to disk before reboot. We have seen both in different runs.
# Connect to A and verify that /test1 does not exist. It exists on B and C.

Slowing down disk syncs:
{noformat}
echo 360000 | sudo tee /proc/sys/vm/dirty_writeback_centisecs
echo 360000 | sudo tee /proc/sys/vm/dirty_expire_centisecs
echo 99 | sudo tee /proc/sys/vm/dirty_background_ratio
echo 99 | sudo tee /proc/sys/vm/dirty_ratio
{noformat}
9223372036854775807 No Perforce job exists for this issue. 3 9223372036854775807
3 years, 27 weeks, 4 days ago 0|i2nsfb:
ZooKeeper ZOOKEEPER-2309

TestClient fails

Bug Resolved Blocker Not A Problem Flavio Paiva Junqueira Flavio Paiva Junqueira Flavio Paiva Junqueira 30/Oct/15 11:07   30/Oct/15 12:09 30/Oct/15 12:09         0 1   I'm getting this out of a fresh copy of branch-3.4.

{noformat}
tests/TestClient.cc:375: Assertion: equality assertion failed [Expected: -101, Actual : -4]
tests/TestClient.cc:300: Assertion: assertion failed [Expression: ctx.waitForConnected(zk)]
Failures !!!
{noformat}
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
4 years, 20 weeks, 6 days ago 0|i2nqtj:
ZooKeeper ZOOKEEPER-2308

Warning messages in the zookeeper logs

Bug Open Major Unresolved Unassigned Ramesh Gopal Ramesh Gopal 29/Oct/15 12:57   29/Oct/15 12:57           1 3   telnet localhost 2181 gives the following warning messages.

2015-07-08 09:26:13,785 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@347] - caught end of stream exception
EndOfStreamException: Unable to read additional data from client sessionid 0x34e6a8473084e8d, likely client has closed socket
at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:218)
at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
at java.lang.Thread.run(Thread.java:853)
2015-07-08 09:26:13,785 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@347] - caught end of stream exception
EndOfStreamException: Unable to read additional data from client sessionid 0x34e6a8473084e7f, likely client has closed socket
at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:218)
at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
at java.lang.Thread.run(Thread.java:853)
2015-07-08 09:26:13,813 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@347] - caught end of stream exceptionEndOfStreamException: Unable to read additional data from client sessionid 0x34e6a8473084e8b, likely client has closed socket
at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:218)
at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
at java.lang.Thread.run(Thread.java:853)
2015-07-08 09:26:13,963 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@347] - caught end of stream exceptionEndOfStreamException: Unable to read additional data from client sessionid 0x34e6a8473084e80, likely client has closed socket
at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:218)
at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
at java.lang.Thread.run(Thread.java:853)
2015-07-08 09:26:13,980 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@347] - caught end of stream exceptionEndOfStreamException: Unable to read additional data from client sessionid 0x34e6a8473084e81, likely client has closed socket
at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:218)
at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
at java.lang.Thread.run(Thread.java:853)
2015-07-08 09:26:13,982 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@347] - caught end of stream exceptionEndOfStreamException: Unable to read additional data from client sessionid 0x34e6a8473084e7c, likely client has closed socket
at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:218)
at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
at java.lang.Thread.run(Thread.java:853)
2015-07-08 09:26:18,453 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@347] - caught end of stream exceptionEndOfStreamException: Unable to read additional data from client sessionid 0x34e6a8473084e84, likely client has closed socket
at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:218)
at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
at java.lang.Thread.run(Thread.java:853)
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
4 years, 21 weeks ago 0|i2np7b:
ZooKeeper ZOOKEEPER-2307

ZooKeeper not starting because acceptedEpoch is less than the currentEpoch

Bug Resolved Major Fixed Mohammad Arshad Mohammad Arshad Mohammad Arshad 29/Oct/15 11:12   03/Feb/20 05:35 17/Dec/19 07:48   3.6.0 server   1 12 0 7800   This issue occurred in one of our test environment where disk was being changed to read only very frequently.
The the scenario is as follows:
# Configure three node ZooKeeper cluster, lets say nodes are A, B and C
# Start A and B. Both A and B start successfully, quorum is running.
# Start C, because of IO error C fails to update acceptedEpoch file. But C also starts successfully, joins the quorum as follower
# Stop C
# Start C, bellow exception with message "The accepted epoch, 0 is less than the current epoch, 1" is thrown
{code}
2015-10-29 16:52:32,942 [myid:3] - ERROR [main:QuorumPeer@784] - Unable to load database on disk
java.io.IOException: The accepted epoch, 0 is less than the current epoch, 1
at org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase(QuorumPeer.java:781)
at org.apache.zookeeper.server.quorum.QuorumPeer.start(QuorumPeer.java:720)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:202)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:139)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:88)
2015-10-29 16:52:32,946 [myid:3] - ERROR [main:QuorumPeerMain@111] - Unexpected exception, exiting abnormally
java.lang.RuntimeException: Unable to run quorum server
at org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase(QuorumPeer.java:785)
at org.apache.zookeeper.server.quorum.QuorumPeer.start(QuorumPeer.java:720)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:202)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:139)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:88)
Caused by: java.io.IOException: The accepted epoch, 0 is less than the current epoch, 1
at org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase(QuorumPeer.java:781)
{code}
100% 100% 7800 0 pull-request-available 9223372036854775807 No Perforce job exists for this issue. 4 9223372036854775807
13 weeks, 2 days ago 0|i2nozz:
ZooKeeper ZOOKEEPER-2306

Remove file delete duplicate code from test code

Improvement Closed Major Fixed Mohammad Arshad Mohammad Arshad Mohammad Arshad 28/Oct/15 03:41   21/Jul/16 16:18 06/Dec/15 15:32   3.5.2, 3.6.0 tests   0 3   Code to delete a folder recursive is written in multiple test files. Following are the files containing the same piece of code.
{code}
src/java/systest/org/apache/zookeeper/test/system/QuorumPeerInstance.java
src/java/test/org/apache/zookeeper/test/ClientBase.java
src/java/test/org/apache/zookeeper/server/quorum/LearnerTest.java
src/java/test/org/apache/zookeeper/server/quorum/Zab1_0Test.java
{code}
Remove duplicate code from these files
9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
4 years, 15 weeks, 4 days ago 0|i2nm8v:
ZooKeeper ZOOKEEPER-2305

created

Bug Open Major Unresolved Unassigned shengfeng shengfeng 27/Oct/15 07:48   28/Oct/15 01:45   3.4.6   quorum   0 1   deleteChild method in Pathtrie class, childNode.getChildren().length == 1, why not childNode.getChildren().length == 0 ? 9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
4 years, 21 weeks, 2 days ago 0|i2nklr:
ZooKeeper ZOOKEEPER-2304

JMX ClientPort from ZooKeeperServerBean incorrect

Bug Closed Major Fixed Mohammad Arshad Mohammad Arshad Mohammad Arshad 26/Oct/15 07:47   21/Jul/16 16:18 06/Dec/15 15:39   3.5.2 jmx   0 4   The "ClientPort" property of {{org.apache.zookeeper.server.ZooKeeperServerBean}} returns incorrect value. It includes address also like 192.168.1.2:2183. It should return only port 9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
4 years, 15 weeks, 4 days ago 0|i2nio7:
ZooKeeper ZOOKEEPER-2303

Zookeeper fails to compile (mesos driver) on Raspberry Pi2

Bug Resolved Critical Not A Problem Steven Fisher Steven Fisher Steven Fisher 25/Oct/15 11:46   27/Oct/15 12:18 27/Oct/15 12:17 3.4.3, 3.4.6   build   0 2   Raspberry Pi - Linux rpi2 3.18.6-v7+ #3 SMP PREEMPT Mon Feb 9 15:39:54 UTC 2015 armv7l GNU/Linux
Trying to compile mesos on Raspberry Pi 2. Zookeeper builds ok with ant directly (Java only), but when using make for mesos it trys to compile the C code and this fails with the error:
libtool: compile: gcc -DHAVE_CONFIG_H -I. -I./include -I./tests -I./generated -DTHREADED -g -O2 -D_GNU_SOURCE -MT libzkmt_la-mt_adaptor.lo -MD -MP -MF .deps/libzkmt_la-mt_adaptor.Tpo -c src/mt_adaptor.c -fPIC -DPIC -o libzkmt_la-mt_adaptor.o
/tmp/ccw07Ju5.s: Assembler messages:
/tmp/ccw07Ju5.s:1515: Error: bad instruction `lock xaddl r1,[r0]'
Makefile:823: recipe for target 'libzkmt_la-mt_adaptor.lo' failed
make[5]: *** [libzkmt_la-mt_adaptor.lo] Error 1

The memos release comes with 3.4.5 but I have also tried 3.4.6
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
4 years, 21 weeks, 2 days ago 0|i2nhtz:
ZooKeeper ZOOKEEPER-2302

Some test cases are not running because wrongly named

Bug Closed Major Fixed Mohammad Arshad Mohammad Arshad Mohammad Arshad 25/Oct/15 06:52   21/Jul/16 16:18 28/Oct/15 17:58   3.5.2, 3.6.0 tests   0 5   When run ZooKeeper test cases following two test classes never run because wrong naming convention is followed.
{code}
org.apache.zookeeper.server.quorum.TestQuorumPeerConfig
org.apache.zookeeper.server.quorum.TestRemotePeerBean
{code}

Name of these test classes should be changed to
{code}
org.apache.zookeeper.server.quorum.QuorumPeerConfigTest
org.apache.zookeeper.server.quorum.RemotePeerBeanTest
{code}

9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
4 years, 21 weeks, 1 day ago 0|i2nhof:
ZooKeeper ZOOKEEPER-2301

QuorumPeer does not listen on passed client IP in the constructor

Bug Closed Major Fixed Mohammad Arshad Mohammad Arshad Mohammad Arshad 25/Oct/15 05:44   21/Jul/16 16:18 07/Dec/15 00:01   3.5.2 server   0 4   QuorumPeer does not listen on passed client IP in the constructor, for client connection. It always listens on all IP(0.0.0.0 or 0:0:0:0:0:0:0:0). This happens only when QuorumPeer is created using any of the bellow constructors
{code}
org.apache.zookeeper.server.quorum.QuorumPeer.QuorumPeer(Map<Long,QuorumServer> quorumPeers, File snapDir,
File logDir, int clientPort, int electionAlg,
long myid, int tickTime, int initLimit, int syncLimit)
{code}
{code}
org.apache.zookeeper.server.quorum.QuorumPeer.QuorumPeer(Map<Long,QuorumServer> quorumPeers, File snapDir,
File logDir, int clientPort, int electionAlg,
long myid, int tickTime, int initLimit, int syncLimit,
QuorumVerifier quorumConfig)
{code}
9223372036854775807 No Perforce job exists for this issue. 3 9223372036854775807
4 years, 15 weeks, 3 days ago 0|i2nhmv:
ZooKeeper ZOOKEEPER-2300

Expose SecureClientPort and SecureClientAddress JMX properties

Improvement Closed Major Fixed Mohammad Arshad Mohammad Arshad Mohammad Arshad 25/Oct/15 03:07   21/Jul/16 16:18 07/Dec/15 00:28 3.5.0 3.5.2 jmx   0 3   ZooKeeper currently exposes ClientPort and ClientAddress JMX properties.
Same way we should expose SecureClientPort and SecureClientAddress.
The values for these two properties will be decided based on the configured value of secureClientPort and secureClientPortAddress

The value of SecureClientPort will be:
secureClientPort {color:blue}(if secureClientPort is configured){color}
empty string {color:blue}(If secureClientPort is not configured){color}

The value of SecureClientAddress will be:
secureClientPortAddress:secureClientPort {color:blue}(if both secureClientPort and secureClientPortAddress are configured){color}
0.0.0.0:secureClientPort or 0:0:0:0:0:0:0:0:secureClientPort {color:blue}(if only secureClientPort is configured){color}
empty string {color:blue}(If secureClientPort is not configured){color}
9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
4 years, 15 weeks, 3 days ago 0|i2nhkn:
ZooKeeper ZOOKEEPER-2299

NullPointerException in LocalPeerBean for ClientAddress

Bug Closed Major Fixed Mohammad Arshad Mohammad Arshad Mohammad Arshad 22/Oct/15 05:28   21/Jul/16 16:18 07/Dec/15 00:08   3.5.2, 3.6.0 jmx   0 6   When clientPortAddress is not configured LocalPeerBean throws NullPointerException.

*Expected Behavior:*
# When only clientPort is configured ClientAddress value should be 0.0.0.0:clientPort or 0:0:0:0:0:0:0:0:clientPort
# When both clientPort clientPortAddress are configured then expected value is clientPortAddress:clientPort
9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
4 years, 15 weeks, 3 days ago 0|i2nddb:
ZooKeeper ZOOKEEPER-2298

zookeeper: Should retry on EAI_NONAME return from getaddrinfo()

Bug Resolved Minor Invalid Unassigned Neil Conway Neil Conway 22/Oct/15 00:31   22/Oct/15 00:33 22/Oct/15 00:33         0 1   The zookeeper interface is designed to retry (once per second for up to ten minutes) if one or more of the Zookeeper hostnames can't be resolved (see [MESOS-1326] and [MESOS-1523]).

However, the current implementation assumes that a DNS resolution failure is indicated by zookeeper_init() returning NULL and errno being set to EINVAL (Zk translates getaddrinfo() failures into errno values). However, the current Zk code does:

{code}
static int getaddrinfo_errno(int rc) {
switch(rc) {
case EAI_NONAME:
// ZOOKEEPER-1323 EAI_NODATA and EAI_ADDRFAMILY are deprecated in FreeBSD.
#if defined EAI_NODATA && EAI_NODATA != EAI_NONAME
case EAI_NODATA:
#endif
return ENOENT;
case EAI_MEMORY:
return ENOMEM;
default:
return EINVAL;
}
}
{code}

getaddrinfo() returns EAI_NONAME when "the node or service is not known"; per discussion in [MESOS-2186], this seems to happen intermittently due to DNS failures.

Proposed fix: looking at errno is always going to be somewhat fragile, but if we're going to continue doing that, we should check for ENOENT as well as EINVAL.
mesosphere 9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
4 years, 22 weeks ago 0|i2nd0f:
ZooKeeper ZOOKEEPER-2297

NPE is thrown while creating "key manager" and "trust manager"

Bug Closed Blocker Fixed Mohammad Arshad Anushri Anushri 19/Oct/15 07:38   21/Jul/16 16:18 23/Jun/16 13:43 3.5.1 3.5.2, 3.6.0 server   0 10   Suse 11 sp 3 NPE is thrown while creating "key manager" and "trust manager" , even though the zk setup is in non-secure mode

bq. 2015-10-19 12:54:12,278 [myid:2] - ERROR [ProcessThread(sid:2 cport:-1)::X509AuthenticationProvider@78] - Failed to create key manager

bq. org.apache.zookeeper.common.X509Exception$KeyManagerException: java.lang.NullPointerException

at org.apache.zookeeper.common.X509Util.createKeyManager(X509Util.java:129)

at org.apache.zookeeper.server.auth.X509AuthenticationProvider.<init>(X509AuthenticationProvider.java:75)

at org.apache.zookeeper.server.auth.ProviderRegistry.initialize(ProviderRegistry.java:42)

at org.apache.zookeeper.server.auth.ProviderRegistry.getProvider(ProviderRegistry.java:68)

at org.apache.zookeeper.server.PrepRequestProcessor.fixupACL(PrepRequestProcessor.java:952)

at org.apache.zookeeper.server.PrepRequestProcessor.pRequest2Txn(PrepRequestProcessor.java:379)

at org.apache.zookeeper.server.PrepRequestProcessor.pRequest(PrepRequestProcessor.java:716)

at org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:144)

Caused by: java.lang.NullPointerException

at org.apache.zookeeper.common.X509Util.createKeyManager(X509Util.java:113)

... 7 more

bq. 2015-10-19 12:54:12,279 [myid:2] - ERROR [ProcessThread(sid:2 cport:-1)::X509AuthenticationProvider@90] - Failed to create trust manager

bq. org.apache.zookeeper.common.X509Exception$TrustManagerException: java.lang.NullPointerException

at org.apache.zookeeper.common.X509Util.createTrustManager(X509Util.java:158)

at org.apache.zookeeper.server.auth.X509AuthenticationProvider.<init>(X509AuthenticationProvider.java:87)

at org.apache.zookeeper.server.auth.ProviderRegistry.initialize(ProviderRegistry.java:42)

at org.apache.zookeeper.server.auth.ProviderRegistry.getProvider(ProviderRegistry.java:68)

at org.apache.zookeeper.server.PrepRequestProcessor.fixupACL(PrepRequestProcessor.java:952)

at org.apache.zookeeper.server.PrepRequestProcessor.pRequest2Txn(PrepRequestProcessor.java:379)

at org.apache.zookeeper.server.PrepRequestProcessor.pRequest(PrepRequestProcessor.java:716)

at org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:144)

Caused by: java.lang.NullPointerException

at org.apache.zookeeper.common.X509Util.createTrustManager(X509Util.java:143)

... 7 more

9223372036854775807 No Perforce job exists for this issue. 8 9223372036854775807
3 years, 39 weeks ago 0|i2n6nz:
ZooKeeper ZOOKEEPER-2296

compilation broken for 3.4

Bug Resolved Blocker Fixed Raúl Gutiérrez Segalés Raúl Gutiérrez Segalés Raúl Gutiérrez Segalés 18/Oct/15 17:08   21/Oct/15 18:12 21/Oct/15 18:12   3.4.7     0 3   Apparently, ZOOKEEPER-2253 wasn't fully backported from trunk so it doesn't compile now.

We should make sure jenkins runs for 3.4, to catch these issues in the future.
9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
4 years, 22 weeks, 1 day ago
Reviewed
0|i2n61b:
ZooKeeper ZOOKEEPER-2295

TGT refresh time logic is wrong

Bug Closed Major Fixed Mohammad Arshad Mohammad Arshad Mohammad Arshad 17/Oct/15 02:32   21/Jul/16 16:18 08/Dec/15 23:11 3.4.7, 3.5.1 3.4.8, 3.5.2, 3.6.0     0 4   When Kerberos is used as authentication mechanism some time TGT is getting expired because it is not refreshed timely.
The scenario is as follow:
suppose now=8 (the current milliseconds)
next refresh time= 10
TGT expire time= 9

*Current behaviour:* Error is logged and TGT refresh thread exits.
*Expected behaviour:* TGT should be refreshed immediately(now) instead of nextRefreshTime
9223372036854775807 No Perforce job exists for this issue. 3 9223372036854775807
4 years, 15 weeks, 1 day ago 0|i2n5db:
ZooKeeper ZOOKEEPER-2294

Ant target generate-clover-reports is broken

Bug Closed Major Fixed Charlie Helin Charlie Helin Charlie Helin 14/Oct/15 15:50   21/Jul/16 16:18 03/Mar/16 12:27   3.4.9, 3.5.2, 3.6.0 build   0 4   It appears that the current implementation of 'generate-clover-reports' is broken.

# It doesn't define the clover-report task
# The clover-report element is missing the proper clover db reference
9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
4 years, 3 weeks ago
Reviewed
0|i2n0vj:
ZooKeeper ZOOKEEPER-2293

Incorrect log level 'warn' is resulting in clutering of logs, Suggest to demote it to DEBUG from WARN

Bug Open Minor Unresolved Rabi Kumar K C Praveena Manvi Praveena Manvi 14/Oct/15 03:56   01/Feb/20 12:19   3.4.6   server   0 4 0 1800   In https://svn.apache.org/repos/asf/zookeeper/trunk/src/java/main/org/apache/zookeeper/server/ZooKeeperServer.java
If readOnly flag is not being sent it gets logged as warning. Since we have enabled warning, the server gets filled up with messages like

Btw, readonly is optional and introduced later (http://wiki.apache.org/hadoop/ZooKeeper/GSoCReadOnlyMode),

{code}
015-08-14T11:03:11+00:00 Connection request from old client /192.168.24.16:14479; will be dropped if server is in r-o mode
...
2015-08-14T11:21:56+00:00 Connection request from old client
2015-08-14T11:18:40+00:00 Connection request from old client /192.168.24.14:12135; will be dropped if server is in r-o mode
2015-08-14T11:19:40+00:00 Connection request from old client /192.168.24.14:12310; will be dropped if server is in r-o mode
{code}

we are just forced to send read-only flag which is optional to avoid wrong logging level chosen by zookeeper.
{code}
boolean readOnly = false;
try {
readOnly = bia.readBool("readOnly");
cnxn.isOldClient = false;
} catch (IOException e) {
// this is ok -- just a packet from an old client which
// doesn't contain readOnly field
LOG.warn("Connection request from old client "
+ cnxn.getRemoteSocketAddress()
+ "; will be dropped if server is in r-o mode");
}
{code}

Suggest to demote the same to DEBUG as its not intended to warn in anyway.

100% 100% 1800 0 pull-request-available 9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
6 weeks, 5 days ago 0|i2mzn3:
ZooKeeper ZOOKEEPER-2292

Sign the download package

Improvement Resolved Major Duplicate Chris Nauroth Elias Levy Elias Levy 13/Oct/15 19:12   14/Oct/15 11:47 14/Oct/15 11:47     build   0 2   Current ZK is made available for download as a compressed archive. Within the archive, there is a cryptographic signature for the ZK JAR file. Alas, the signature does not cover any of the other executable components that ZK depends on, such as JARs in the lib directory or the scripts in the bin directory. These could be tampered with.

The whole download package should be signed and the signature made available along with it.
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
4 years, 23 weeks, 1 day ago 0|i2mz5j:
ZooKeeper ZOOKEEPER-2291

JMX log should not be printed while stopping ZK Server

Bug Open Minor Unresolved Neha Bathra Neha Bathra Neha Bathra 13/Oct/15 11:41   14/Oct/15 02:37           0 2   While stopping zookeeper prints below message :

"ZooKeeper JMX enabled by default
ZooKeeper remote JMX Port set to 2022
ZooKeeper remote JMX authenticate set to false
ZooKeeper remote JMX ssl set to false
ZooKeeper remote JMX log4j set to true
Stopping zookeeper ...STOPPED"

JMX message should only be printed while starting ZK.
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
4 years, 23 weeks, 2 days ago 0|i2myfb:
ZooKeeper ZOOKEEPER-2290

Add read/write qps metrics in monitor cmd

Improvement Patch Available Minor Unresolved Shaohui Liu Shaohui Liu Shaohui Liu 11/Oct/15 23:40   14/Dec/19 06:07   3.4.6 3.7.0     0 4   Read/write qps are important metrics to show the pressure of the cluster. We can also use it to alert about some abuse of zookeeper. monitor 9223372036854775807 No Perforce job exists for this issue. 5 9223372036854775807
4 years, 6 days ago 0|i2mvcv:
ZooKeeper ZOOKEEPER-2289

Support use of SASL "auth-int" and "auth-conf" quality of protection.

New Feature Open Major Unresolved Unassigned Chris Nauroth Chris Nauroth 10/Oct/15 18:50   16/Oct/15 18:46       java client, server   1 4   The current codebase supports use of SASL for authenticating connections, but it does not allow specifying the desired SASL Quality of Protection (QOP). It always uses the default QOP, which is "auth" (authentication only). This issue proposes to support the full set of available QOP settings by adding support for "auth-int" (authentication and integrity) and "auth-conf" (authentication and integrity and privacy/encryption). 9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
4 years, 23 weeks, 4 days ago 0|i2muvr:
ZooKeeper ZOOKEEPER-2288

During shutdown, server may fail to ack completed transactions to clients.

Bug Patch Available Major Unresolved Chris Nauroth Chris Nauroth Chris Nauroth 10/Oct/15 18:16   10/Oct/15 18:55       server   0 2   During shutdown, requests may still be in flight in the request processing pipeline. Some of these requests have reached a state where the transaction has executed and committed, but has not yet been acknowledged back to the client. It's possible that these transactions will not ack to the client before the shutdown sequence completes. 9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
4 years, 23 weeks, 5 days ago 0|i2muvb:
ZooKeeper ZOOKEEPER-2287

Audit logging the zookeeper operations

New Feature Resolved Major Duplicate Mohammad Arshad nijel nijel 30/Sep/15 07:17   29/Jun/16 15:01 29/Jun/16 15:01         0 3   As of now Zookeeper does not support auditing the user operations
This is a very important tracability in distributed cluster to trace the operations

We can have a separate logger and log file.
Can start with normal node change operations.

Please share your thoughts ?
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
4 years, 25 weeks, 1 day ago 0|i2mg7r:
ZooKeeper ZOOKEEPER-2286

Doc Issue

Improvement Open Minor Unresolved Unassigned richard.zhao richard.zhao 29/Sep/15 03:22   29/Sep/15 03:22   3.4.6   documentation   0 1 60 60 0% First, thanks for your detail documents. It's very helpful.

In this one "Programming with ZooKeeper - A basic tutorial"
(URL: https://zookeeper.apache.org/doc/trunk/zookeeperTutorial.html)

The demo code has a small issue.
---------------------------------------------------------------------------------
boolean enter() throws KeeperException, InterruptedException{
zk.create(root + "/" + name, new byte[0], Ids.OPEN_ACL_UNSAFE,
CreateMode.EPHEMERAL_SEQUENTIAL);
while (true) {
synchronized (mutex) {
List<String> list = zk.getChildren(root, true);

if (list.size() < size) {
mutex.wait();
} else {
return true;
}
}
}
}
---------------------------------------------------------------------------------

The invocation of zk.create() should be under the if() branch like bellow
---------------------------------------------------------------------------------
if (list.size() < size) {
zk.create(root + "/" + name, new byte[0],
Ids.OPEN_ACL_UNSAFE,
CreateMode.EPHEMERAL_SEQUENTIAL);

mutex.wait();
} else {
---------------------------------------------------------------------------------

The function leave() has a similar problem. And the invocation of zk.delete() should be as follows
---------------------------------------------------------------------------------
zk.delete(root + "/" + list.get(0), 0);
---------------------------------------------------------------------------------

Hope it can help other doc readers.
0% 0% 60 60 9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
4 years, 25 weeks, 2 days ago 0|i2me7r:
ZooKeeper ZOOKEEPER-2285

QuorumTest's ignored test case causes wrong CI pre-commit feedback

Bug Patch Available Major Unresolved Mohammad Arshad Mohammad Arshad Mohammad Arshad 28/Sep/15 17:25   05/Feb/20 07:11   3.5.0 3.7.0, 3.5.8 tests   0 4   # test case {{org.apache.zookeeper.test.QuorumTest.testSessionMove()}} is marked ignored by ZOOKEEPER-907
# Most of the CI pre-commit feedback is -1 because of above ignored test case.
# Test case is locally passing

The ignore tag should be removed from testSessionMove test case.
9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
3 years, 33 weeks, 6 days ago 0|i2mdnb:
ZooKeeper ZOOKEEPER-2284

LogFormatter and SnapshotFormatter does not handle FileNotFoundException gracefully

Bug Closed Minor Fixed maoling Mohammad Arshad Mohammad Arshad 28/Sep/15 14:27   20/May/19 13:51 22/Feb/19 04:45 3.5.0 3.6.0, 3.5.5     0 6 0 12600   {{LogFormatter}} and {{SnapshotFormatter}} does not handle FileNotFoundException gracefully. If file no exist then these classes propagate the exception to console.
{code}
Exception in thread "main" java.io.FileNotFoundException: log.1 (The system cannot find the file specified)
at java.io.FileInputStream.open(Native Method)
at java.io.FileInputStream.<init>(FileInputStream.java:146)
at java.io.FileInputStream.<init>(FileInputStream.java:101)
at org.apache.zookeeper.server.LogFormatter.main(LogFormatter.java:49)
{code}

File existence should be validated and appropriate message should be displayed on console if file does not exist
100% 100% 12600 0 pull-request-available 9223372036854775807 No Perforce job exists for this issue. 4 9223372036854775807
1 year, 9 weeks, 3 days ago 0|i2mdcv:
ZooKeeper ZOOKEEPER-2283

traceFile property is not used in the ZooKeeper, it should be removed from documentation

Bug Closed Major Fixed Mohammad Arshad Mohammad Arshad Mohammad Arshad 28/Sep/15 13:47   09/May/19 14:59 14/Mar/16 03:00 3.4.8, 3.5.0 3.4.9, 3.5.2, 3.6.0 documentation   0 4   zookeeperAdmin guide have following description for traceFile property
{noformat}
traceFile
(Java system property: requestTraceFile)
If this option is defined, requests will be will logged to a trace file named traceFile.year.month.day.
Use of this option provides useful debugging information, but will impact performance. (Note: The system property has no zookeeper prefix, and the configuration variable name is different from the system property. Yes - it's not consistent, and it's annoying.)
{noformat}
But this property is used no where in the whole ZooKeeper code. it should be removed from documentation

9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
4 years, 1 week, 3 days ago
Reviewed
0|i2md8v:
ZooKeeper ZOOKEEPER-2282

chroot not stripped from path in asynchronous callbacks

Bug Closed Critical Fixed Andrew Grasso Andrew Grasso Andrew Grasso 28/Sep/15 13:19   14/Feb/20 10:23 27/Sep/19 09:43 3.4.6, 3.5.0 3.6.0, 3.5.7 c client   1 5 3600 0 4200 116% Centos 6.6 Callbacks passed to [zoo_acreate], [zoo_async], and [zoo_amulti] (for create ops) are called on paths that include the chroot. This is analagous to issue 1027, which fixed this bug for synchronous calls.
I've created a patch to fix this in trunk
100% 100% 4200 0 3600 pull-request-available 9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
24 weeks, 6 days ago 0|i2md6n:
ZooKeeper ZOOKEEPER-2281

ZK Server startup fails if there are spaces in the JAVA_HOME path

Bug Closed Minor Fixed Neha Bathra Neha Bathra Neha Bathra 23/Sep/15 10:43   21/Jul/16 16:18 09/Dec/15 18:27   3.4.8, 3.5.2, 3.6.0 scripts   1 8   Windows Zookeeper startup fails if there are spaces in the %JAVA_HOME% variable.
{code}
if not exist %JAVA_HOME%\bin\java.exe (
echo Error: JAVA_HOME is incorrectly set.
goto :eof
)

set JAVA=%JAVA_HOME%\bin\java
{code}
9223372036854775807 No Perforce job exists for this issue. 3 9223372036854775807
4 years, 15 weeks, 1 day ago
Reviewed
0|i2lgmn:
ZooKeeper ZOOKEEPER-2280

NettyServerCnxnFactory doesn't honor maxClientCnxns param

Bug Open Major Unresolved Unassigned Edward Ribeiro Edward Ribeiro 21/Sep/15 21:13   05/Feb/20 07:16   3.4.6, 3.5.0, 3.5.1 3.7.0, 3.5.8 server   0 7 0 1800   Even though NettyServerCnxnFactory has maxClientCnxns (default to 60) it doesn't enforce this limit in the code. 100% 100% 1800 0 pull-request-available 9223372036854775807 No Perforce job exists for this issue. 2 9223372036854775807
2 years, 51 weeks, 3 days ago 0|i2ldjj:
ZooKeeper ZOOKEEPER-2279

QuorumPeer loadDataBase() error message is incorrect

Bug Closed Major Fixed Mohammad Arshad sunhaitao sunhaitao 21/Sep/15 11:10   21/Jul/16 16:18 25/Sep/15 02:35 3.5.0, 3.5.1 3.4.7, 3.5.2, 3.6.0 quorum   0 5   in loadDataBase() method, the below info is incorrect.
if (acceptedEpoch < currentEpoch) {
throw new IOException("The current epoch, " + ZxidUtils.zxidToString(currentEpoch) + " is less than the accepted epoch, " + ZxidUtils.zxidToString(acceptedEpoch));
}

It should print:
Change the message to ("The accepted epoch, " + ZxidUtils.zxidToString(acceptedEpoch) + " is less than the current epoch, " + ZxidUtils.zxidToString(currentEpoch)
9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
4 years, 25 weeks, 6 days ago 0|i2lclb:
ZooKeeper ZOOKEEPER-2278

4lw command stmk throws NullPointerException

Bug Resolved Major Duplicate Unassigned Neha Bathra Neha Bathra 18/Sep/15 08:50   18/Sep/15 13:59 18/Sep/15 12:51         0 4   run echo stmk 123 | netcat <hostname> <port> fails with NullPointerException 9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
4 years, 26 weeks, 6 days ago 0|i2ke4n:
ZooKeeper ZOOKEEPER-2277

Zookeeper off-line snapshot and transaction log viewer

Improvement Patch Available Major Unresolved Mohammad Arshad Mohammad Arshad Mohammad Arshad 16/Sep/15 12:29   05/Feb/20 07:12     3.7.0, 3.5.8     0 6   {color:blue}Currently ZooKeeper provides utility java program to view the snapshot and transaction off-line but these are not easy to use, also the content is less understandable {color}
In this improvement task I propose following functionality:
1) add zkTool.sh script to view snapshot and transaction.
Usage: zkTool.sh -COMMAND <transaction/snapshot file>
where COMMAND is one of:
otv off-line transaction viewer, prints ZooKeeper transaction log in text format
osv off-line snapshot viewer, prints ZooKeeper snapshot in text format
2) otv command will give output as:
{noformat}
9/4/15 4:37:04 PM IST session 0x1004d19fe6f0002 cxid 0x00000000000c2c (epoch=0,count=3116) zxid 0x00000100000c49 (epoch=1,count=3145) create path="/4da53875-b471-4ab1-9995-03889e73c0a3/node246",data="Quick brown fox jumps over the lazy dog ",acl{e1{perms="cdrwa",id{scheme="world",id="anyone"}}},ephemeral="true",parentCVersion="8"
{noformat}
It is mostly same as {{org.apache.zookeeper.server.LogFormatter}} with some differences.
* epoch and count are separated from zxid.
* operations type will be written instead of code like createSession instead -10.
* showing data.
* permissions are written in letters perms="cdrwa" instead of perms="31" same as {{org.apache.zookeeper.cli.GetAclCommand}}.
* ephemeral="true" instead of ephemeral="1"
* etc.

3) osv command will give output as:
{code}
/67868d36-8bbf-4a8a-a076-f16810ac10de/node540000000010
cZxid = 0x00000100000265 (epoch=1,count=613)
ctime = Fri Sep 04 16:35:58 IST 2015
mZxid = 0x00000100000265 (epoch=1,count=613)
mtime = Fri Sep 04 16:35:58 IST 2015
pZxid = 0x00000100000265 (epoch=1,count=613)
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x1004d19fe6f0002
dataLength = 40
data = Quick brown fox jumps over the lazy dog
{code}
which is almost same as {{org.apache.zookeeper.server.SnapshotFormatter}}
9223372036854775807 No Perforce job exists for this issue. 2 9223372036854775807
1 year, 45 weeks ago 0|i2k9br:
ZooKeeper ZOOKEEPER-2276

Multi operation failure does not include path in KeeperException

Bug Patch Available Minor Unresolved Mohammad Arshad Neha Bathra Neha Bathra 15/Sep/15 01:54   05/Feb/20 07:11   3.5.0 3.7.0, 3.5.8 java client   0 4   Suse 11 SP3 With normal create operation, the path of the failed node is displayed in KeeperException but this is not the case when create operation is through multi api

9223372036854775807 No Perforce job exists for this issue. 4 9223372036854775807
3 years, 25 weeks, 2 days ago 0|i2k6jr:
ZooKeeper ZOOKEEPER-2275

Fix RPM package creation on recent distros

Bug Resolved Major Won't Fix Hannu Valtonen Hannu Valtonen Hannu Valtonen 14/Sep/15 16:21   03/Mar/16 11:24 03/Mar/16 11:24 3.5.1   build   0 2   Three issues with RPM package building,

The install stage was removing BUILDROOT content:

[rpm] + rm -rf /tmp/zkpython_build_rpm/BUILD

Since BUILD and BUILDROOT are actually the same folder, everything is
removed before being used.

The original fix for this problem was submitted by Cédric Lejeune

http://mail-archives.apache.org/mod_mbox/zookeeper-user/201212.mbox/%3C50D2D481.8010507@pt-consulting.eu%3E

The other two issues that need to be fixed are an invalid argument given to popd and a reference to old redhat RPM packaging scripts.
9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
Patch
4 years, 3 weeks ago * Fix RPM creation on newer distributions 0|i2k5sf:
ZooKeeper ZOOKEEPER-2274

ZooKeeperServerMain is difficult to subclass for unit testing

Improvement Patch Available Major Unresolved Jordan Zimmerman Jordan Zimmerman Jordan Zimmerman 11/Sep/15 13:46   02/Mar/16 21:30   3.5.1   server, tests   0 1   Apache Curator needs a testable version of ZooKeeperServerMain. In the past, Curator has used javassist, reflection, etc. but this is all clumsy. With a few trivial changes, Curator could use ZooKeeperServerMain directly by subclassing. 9223372036854775807 No Perforce job exists for this issue. 2 9223372036854775807
4 years, 3 weeks ago 0|i2k2gv:
ZooKeeper ZOOKEEPER-2273

Uninvited ZK joins the ensemble

Bug Open Major Unresolved Unassigned Benjamin Jaton Benjamin Jaton 10/Sep/15 14:45   16/Sep/15 19:14           0 2   Scenario:
- Install a Zookeeper on machine A
- Install a Zookeeper on machine B, joining A to form an ensemble
- Reinstall ZooKeeper on A (but with standaloneEnabled=false)
-> B automatically joins A to form an ensemble again

I think the work needed is discussed and addressed in ZOOKEEPER-832.
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
4 years, 27 weeks, 1 day ago 0|i2k0i7:
ZooKeeper ZOOKEEPER-2272

Code clean up in ZooKeeperServer and KerberosName

Improvement Patch Available Trivial Unresolved Mohammad Arshad Mohammad Arshad Mohammad Arshad 10/Sep/15 02:36   05/Feb/20 07:11   3.5.0 3.7.0, 3.5.8 server   0 2   # Following code in {{org.apache.zookeeper.server.ZooKeeperServer}} should be cleaned up. Some how it got missed in code review
{code}
if ((System.getProperty("zookeeper.allowSaslFailedClients") != null)
&& (System.getProperty("zookeeper.allowSaslFailedClients").equals("true"))) {
{code}
should be replaced with
{code}
if(Boolean.getBoolean("zookeeper.allowSaslFailedClients"))
{code}
# Similar code clean up can be done in {{org.apache.zookeeper.server.auth.KerberosName}}
9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
3 years, 4 weeks, 1 day ago 0|i2jzdz:
ZooKeeper ZOOKEEPER-2271

ZOOKEEPER-2270 Allow MBeanRegistry to be overridden for better unit tests in 3.4.x

Sub-task Resolved Major Fixed Jordan Zimmerman Jordan Zimmerman Jordan Zimmerman 09/Sep/15 18:03   16/Oct/16 11:30 10/Sep/15 00:17 3.4.6 3.4.7 server   0 2   9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
4 years, 28 weeks ago 0|i2jyu7:
ZooKeeper ZOOKEEPER-2270

Allow MBeanRegistry to be overridden for better unit tests

Improvement Resolved Major Fixed Jordan Zimmerman Jordan Zimmerman Jordan Zimmerman 09/Sep/15 17:45   16/Oct/16 11:30 10/Sep/15 00:18 3.4.6, 3.5.1 3.5.2, 3.6.0 server   0 3   ZOOKEEPER-2271 Apache Curator currently must use byte code re-writing to prevent the MBeanRegistry from polluting the Platform MBeanServer. Provide a simple way to avoid this. 9223372036854775807 No Perforce job exists for this issue. 2 9223372036854775807
4 years, 28 weeks ago 0|i2jysv:
ZooKeeper ZOOKEEPER-2269

NullPointerException in RemotePeerBean

Bug Closed Major Fixed Mohammad Arshad Mohammad Arshad Mohammad Arshad 09/Sep/15 08:16   21/Jul/16 16:18 10/Sep/15 00:50 3.5.0 3.5.2, 3.6.0 jmx   0 4   {code}org.apache.zookeeper.server.quorum.RemotePeerBean.getClientAddress(){code} throws NullPointerException when clientPort is not part of dynamic configuration.

9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
4 years, 28 weeks ago 0|i2jy0v:
ZooKeeper ZOOKEEPER-2268

Zookeeper doc creation fails on windows

Bug Closed Major Fixed Mohammad Arshad Mohammad Arshad Mohammad Arshad 08/Sep/15 10:08   21/Jul/16 16:18 03/Oct/15 17:04 3.5.0 3.4.7, 3.5.2 build   0 4   Zookeeper doc creation fails on windows with following error
{code}
D:\gitHome\zookeeper-trunk\build.xml:484: Execute failed: java.io.IOException: Cannot run program "C:\non-install\apache-forrest-0.9\bin\forrest"
y "D:\gitHome\zookeeper-trunk\src\docs"): CreateProcess error=193, %1 is not a valid Win32 application

{code}
9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
4 years, 24 weeks, 3 days ago
Reviewed
0|i2jw27:
ZooKeeper ZOOKEEPER-2267

zoo_amulti doesn't correctly return ZINVALIDSTATE errors

Improvement Open Minor Unresolved Unassigned Mark Flickinger Mark Flickinger 08/Sep/15 02:28   28/Nov/16 15:18   3.4.0   c client   0 2   zoo_amulti will always return ZMARSHALLINGERROR whenever the zhandle is in an unrecoverable state. ZINVALIDSTATE should probably be returned in these situations.

Preferably the return code from the relevant error should be returned as is. At the very least it would be nice if zoo_amulti first checked is_unrecoverable, the way zoo_awget does. It seems the other async functions are implicitly doing this in the *Request_init calls.
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
4 years, 28 weeks, 2 days ago 0|i2jvgv:
ZooKeeper ZOOKEEPER-2266

Integrate JaCoCo Coverage Library

Improvement Patch Available Minor Unresolved Akihiro Suda Akihiro Suda Akihiro Suda 07/Sep/15 05:23   09/Mar/16 06:31       tests   0 3   I would like to propose integration of [JaCoCo|http://www.eclemma.org/jacoco/] coverage library with ZooKeeper.

h4. Purposes
- To find poorly covered methods, and improve JUnit testcases to cover them
- To estimate causes of flaky testcases (e.g. ZOOKEEPER-2080, ZOOKEEPER-2252, ZOOKEEPER-1868) by comparing reports from succeeded experiments and failed ones (I'm recently interested in how we can systematically realize this.)

h4. Advantages of JaCoCo
- Support recent JDKs (including JDK 8)
- Low overhead
- Released under EPL
-- Note: cobertura has been removed from the code base because it is released under GPL ( ZOOKEEPER-75, http://www.apache.org/legal/resolved.html#category-x )


h4. Usage
{panel}
$ ant test #(plus optionally, -Dtestcase=.. -Dtest.method=..)
$ ant jacoco-report
$ x-www-browser build/test/jacoco/reports/index.html &
{panel}
Example: jacoco-report-example.zip

h4. Possible Future Work
- Integrate to Jenkins buildbot so that we can check coverage after each of builds
9223372036854775807 No Perforce job exists for this issue. 5 9223372036854775807
Patch
4 years, 2 weeks, 1 day ago 0|i2jum7:
ZooKeeper ZOOKEEPER-2265

zookeeper build fails while doing configuration for cppunit test

Bug Open Minor Unresolved Mohammad Arshad Mohammad Arshad Mohammad Arshad 03/Sep/15 10:32   05/Feb/20 07:16   3.5.1 3.7.0, 3.5.8 build   0 2   running {color:red}ant tar{color} gives following error
{code}
D:\gitHome\zookeeper-trunk\build.xml:1432: Execute failed: java.io.IOException: Cannot run program "autoreconf" (in directory "D:\gitHome\zookeeper-trunk\src\c"):
{code}
This error is purely environment error and it can be fixed by installing appropriate software package.
But does it really required to configure the cpp unit as {color:red}ant tar{color} target flow does not run cppunit test cases. Then why to configure?
There should be no cppunit configurations for {color:red}ant tar{color} target flow.
9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
4 years, 3 weeks ago 0|i2jr13:
ZooKeeper ZOOKEEPER-2264

Wrong error message when secureClientPortAddress is configured but secureClientPort is not configured

Bug Closed Minor Fixed Mohammad Arshad Mohammad Arshad Mohammad Arshad 31/Aug/15 09:22   21/Jul/16 16:18 06/Sep/15 14:06   3.5.2, 3.6.0 server   0 4   # Wrong error message when secureClientPortAddress is configured but secureClientPort is not configured.
zookeeper throws IllegalArgumentException with error message {{clientPortAddress is set but clientPort is not set}} but should be {{secureClientPortAddress is set but secureClientPort is not set}}
# There is another problem with the same code.
value is assigned to local variable but null check is done on instance variable so we will never get error message for this scenario.
{code}if (this.secureClientPortAddress != null) {{code}
should be replaced with
{code}if (secureClientPortAddress != null) {{code}
# Above problem is there for clientPort scenario also. So we should replace
{code}if (this.clientPortAddress != null) {{code}
with
{code}if (clientPortAddress != null) {{code}
9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
4 years, 28 weeks, 4 days ago 0|i2jlhr:
ZooKeeper ZOOKEEPER-2263

ZooKeeper server should not start when neither clientPort no secureClientPort is configured

Bug Patch Available Minor Unresolved Mohammad Arshad Mohammad Arshad Mohammad Arshad 28/Aug/15 11:14   05/Feb/20 07:11     3.7.0, 3.5.8     0 3   ZooKeeper server should not start when neither clientPort no secureClientPort is configured.
Without any client port ZooKeeper server can not server any purpose. It should simply return with proper error message
9223372036854775807 No Perforce job exists for this issue. 3 9223372036854775807
3 years, 39 weeks, 2 days ago 0|i2jj9b:
ZooKeeper ZOOKEEPER-2262

Admin commands do not include secure client information

Bug Open Major Unresolved Mohammad Arshad Mohammad Arshad Mohammad Arshad 28/Aug/15 10:56   05/Feb/20 07:16   3.5.0 3.7.0, 3.5.8 server   0 1   Admin commands do not include secure client information
connections, configuration, connection_stat_reset and stats admin should include secure client informations
1) configuration should include the secure client port also
2) connections should include secure connections also
3) connection_stat_reset should also reset secure connection
4) stats command should accumulate both secure and non secure information
JettyAdminServer 9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
4 years, 29 weeks, 6 days ago 0|i2ji5z:
ZooKeeper ZOOKEEPER-2261

When only secureClientPort is configured connections, configuration, connection_stat_reset, and stats admin commands throw NullPointerException

Bug Closed Major Fixed Andor Molnar Mohammad Arshad Mohammad Arshad 28/Aug/15 10:32   20/May/19 13:50 10/Sep/18 18:18 3.5.0 3.6.0, 3.5.5     0 4 0 16200   When only secureClientPort is configured connections, configuration, connection_stat_reset and stats admin commands throw NullPointerException. Here is stack trace one of the connections command.
{code}
java.lang.NullPointerException
at org.apache.zookeeper.server.admin.Commands$ConsCommand.run(Commands.java:177)
at org.apache.zookeeper.server.admin.Commands.runCommand(Commands.java:92)
at org.apache.zookeeper.server.admin.JettyAdminServer$CommandServlet.doGet(JettyAdminServer.java:166)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
{code}
100% 100% 16200 0 pull-request-available 9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
1 year, 27 weeks, 2 days ago 0|i2ji3z:
ZooKeeper ZOOKEEPER-2260

Paginated getChildren call

New Feature Patch Available Major Unresolved Marco P. Marco P. Marco P. 27/Aug/15 20:29   05/Feb/20 07:11   3.4.6, 3.5.0 3.7.0, 3.5.8     4 24 0 600   Add pagination support to the getChildren() call, allowing clients to iterate over children N at the time.

Motivations for this include:
- Getting out of a situation where so many children were created that listing them exceeded the network buffer sizes (making it impossible to recover by deleting)[1]
- More efficient traversal of nodes with large number of children [2]

I do have a patch (for 3.4.6) we've been using successfully for a while, but I suspect much more work is needed for this to be accepted.


[1] https://issues.apache.org/jira/browse/ZOOKEEPER-272
[2] https://issues.apache.org/jira/browse/ZOOKEEPER-282
100% 100% 600 0 api, features, pull-request-available 9223372036854775807 No Perforce job exists for this issue. 2 9223372036854775807
1 year, 17 weeks, 1 day ago 0|i2jh5j:
ZooKeeper ZOOKEEPER-2259

4 letter word commands are slow because these unnecessarily go through Sasl authentication

Bug Open Major Unresolved Unassigned Mohammad Arshad Mohammad Arshad 27/Aug/15 10:46   28/Aug/15 02:29           0 2   4 letter word commands are slow because these commands unnecessarily go through Sasl authentication.
{code}
ZooKeeperSaslServer.<init>(Login) line: 48
NettyServerCnxn.<init>(Channel, ZooKeeperServer, NettyServerCnxnFactory) line: 88
NettyServerCnxnFactory$CnxnChannelHandler.channelConnected(ChannelHandlerContext, ChannelStateEvent) line: 89
NettyServerCnxnFactory$CnxnChannelHandler(SimpleChannelHandler).handleUpstream(ChannelHandlerContext, ChannelEvent) line: 118
DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline$DefaultChannelHandlerContext, ChannelEvent) line: 564
{code}
as per the document 4lw commands are executed as bellow
{{$ echo mntr | nc localhost 2185}} .
Even without passing any authentication information it works fine.

So 4lw command either should do authentication properly or it should not go through Sasl authentication flow.



9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
4 years, 30 weeks ago 0|i2jgan:
ZooKeeper ZOOKEEPER-2258

ZooKeeper bin scripts for the Windows needs to be updated

Improvement Open Major Unresolved Unassigned Rakesh Radhakrishnan Rakesh Radhakrishnan 27/Aug/15 04:09   27/Aug/15 04:09       scripts   0 3   The idea of this jira is to port all the changes done to the {{zkxx.sh}} unix based scripts to the Windows {{zkxx.cmd}} scripts. 9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
4 years, 30 weeks ago 0|i2jfr3:
ZooKeeper ZOOKEEPER-2257

Make zookeeper server principal configurable at zookeeper client side

Improvement Resolved Major Duplicate Mohammad Arshad Mohammad Arshad Mohammad Arshad 21/Aug/15 11:25   03/Jan/17 18:48 03/Sep/15 10:45         0 6   Currently Zookeeper client expects zookeeper server's principal to be in the form of zookeeper.sasl.client.username/server-ip for example zookeeper/192.162.1.100.
But this may not always be the case server principal can be some thing like zookeeper/hadoop.foo.com
It would be better if we can make server principal configurable.

Current Code:
{code}
String principalUserName = System.getProperty(ZK_SASL_CLIENT_USERNAME, "zookeeper");
zooKeeperSaslClient = new ZooKeeperSaslClient(principalUserName + "/" + addr.getHostString());
{code}

Proposed Code:
{code}
String serverPrincipal = System.getProperty("zookeeper.server.principal");
if (null != serverPrincipal) {
zooKeeperSaslClient = new ZooKeeperSaslClient(serverPrincipal);
} else {
String principalUserName = System.getProperty(ZK_SASL_CLIENT_USERNAME, "zookeeper");
zooKeeperSaslClient = new ZooKeeperSaslClient(principalUserName + "/" + addr.getHostString());
}
{code}
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 11 weeks, 2 days ago 0|i2j7sf:
ZooKeeper ZOOKEEPER-2256

Zookeeper is not using specified JMX port in zkEnv.sh

Bug Closed Minor Fixed Mohammad Arshad Mohammad Arshad Mohammad Arshad 20/Aug/15 08:57   21/Jul/16 16:18 06/Sep/15 13:51 3.5.0 3.4.7, 3.5.2, 3.6.0 scripts   0 4   Zookeeper is not using specified JMX port.
I put bellow entry in zkEnv.sh
{{export JMXPORT=12345}}
But zookeeper still uses random port for jmx.
9223372036854775807 No Perforce job exists for this issue. 2 9223372036854775807
4 years, 28 weeks, 4 days ago 0|i2j5nj:
ZooKeeper ZOOKEEPER-2255

Use static member classes when permitted

Improvement Patch Available Minor Unresolved Yvonne Ironberg Yvonne Ironberg Yvonne Ironberg 20/Aug/15 01:52   30/Jan/19 07:57           0 2 0 600   Using static member classes saves time and space because an instances of a nonstatic member class has a reference to its enclosing instance.

Also did some style improvements:
- JLS recommends modifiers be in this order: public protected private abstract static final transient volatile synchronized native strictfp.
- Inserted some spaces.
100% 100% 600 0 pull-request-available 9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
4 years, 3 weeks ago 0|i2j53r:
ZooKeeper ZOOKEEPER-2254

CDH 5.4.4 HBASE 1.0.0 with PHX 4.5.0

Bug Resolved Major Won't Fix Unassigned JUAN TORRES JUAN TORRES 19/Aug/15 18:32   20/Aug/15 07:47 20/Aug/15 07:47         0 1   Hi,

I need support. I have tried to troubleshoot this problem, but anything I do, is not working. I do not have a problem with the NN, ZK or any network limitation, but the phoenix still not able to connect the zookeeper servers.

[root@svrdn174 bin]# /opt/phoenix/bin/sqlline.py svrzj001,svrzj002,svrzj003= :2181/hbase Setting property: [isolation, TRANSACTION_READ_COMMITTED]
issuing: !connect jdbc:phoenix:svrzj001,svrzj002,svrzj003:2181/hbase none n= one org.apache.phoenix.jdbc.PhoenixDriver
Connecting to jdbc:phoenix:svrzj001,svrzj002,svrzj003:2181/hbase
15/08/19 18:15:30 WARN util.NativeCodeLoader: Unable to load native-hadoop = library for your platform... using builtin-java classes where applicable
Error: ERROR 103 (08004): Unable to establish connection. (state=3D08004,co=
de=3D103)
java.sql.SQLException: ERROR 103 (08004): Unable to establish connection.
at org.apache.phoenix.exception.SQLExceptionCode$Factory$1.newExcep=
tion(SQLExceptionCode.java:388)
at org.apache.phoenix.exception.SQLExceptionInfo.buildException(SQL=
ExceptionInfo.java:145)
at org.apache.phoenix.query.ConnectionQueryServicesImpl.openConnect=
ion(ConnectionQueryServicesImpl.java:297)
at org.apache.phoenix.query.ConnectionQueryServicesImpl.access$300(=
ConnectionQueryServicesImpl.java:180)
at org.apache.phoenix.query.ConnectionQueryServicesImpl$12.call(Con=
nectionQueryServicesImpl.java:1901)
at org.apache.phoenix.query.ConnectionQueryServicesImpl$12.call(Con=
nectionQueryServicesImpl.java:1880)
at org.apache.phoenix.util.PhoenixContextExecutor.call(PhoenixConte=
xtExecutor.java:77)
at org.apache.phoenix.query.ConnectionQueryServicesImpl.init(Connec=
tionQueryServicesImpl.java:1880)
at org.apache.phoenix.jdbc.PhoenixDriver.getConnectionQueryServices=
(PhoenixDriver.java:180)
at org.apache.phoenix.jdbc.PhoenixEmbeddedDriver.connect(PhoenixEmb=
eddedDriver.java:132)
at org.apache.phoenix.jdbc.PhoenixDriver.connect(PhoenixDriver.java=
:151)
at sqlline.DatabaseConnection.connect(DatabaseConnection.java:157)
at sqlline.DatabaseConnection.getConnection(DatabaseConnection.java=
:203)
at sqlline.Commands.connect(Commands.java:1064)
at sqlline.Commands.connect(Commands.java:996)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessor=
Impl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethod=
AccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at sqlline.ReflectiveCommandHandler.execute(ReflectiveCommandHandle=
r.java:36)
at sqlline.SqlLine.dispatch(SqlLine.java:804)
at sqlline.SqlLine.initArgs(SqlLine.java:588)
at sqlline.SqlLine.begin(SqlLine.java:656)
at sqlline.SqlLine.start(SqlLine.java:398)
at sqlline.SqlLine.main(SqlLine.java:292)
Caused by: java.io.IOException: java.lang.reflect.InvocationTargetException
at org.apache.hadoop.hbase.client.ConnectionFactory.createConnectio=
n(ConnectionFactory.java:240)
at org.apache.hadoop.hbase.client.ConnectionManager.createConnectio=
n(ConnectionManager.java:410)
at org.apache.hadoop.hbase.client.ConnectionManager.createConnectio=
nInternal(ConnectionManager.java:319)
at org.apache.hadoop.hbase.client.HConnectionManager.createConnecti=
on(HConnectionManager.java:144)
at org.apache.phoenix.query.HConnectionFactory$HConnectionFactoryIm=
pl.createConnection(HConnectionFactory.java:47)
at org.apache.phoenix.query.ConnectionQueryServicesImpl.openConnect=
ion(ConnectionQueryServicesImpl.java:295)
... 22 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Me=
thod)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeCons=
tructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Delega=
tingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at org.apache.hadoop.hbase.client.ConnectionFactory.createConnectio=
n(ConnectionFactory.java:238)
... 27 more
Caused by: java.lang.ExceptionInInitializerError
at org.apache.hadoop.hbase.ClusterId.parseFrom(ClusterId.java:64)
at org.apache.hadoop.hbase.zookeeper.ZKClusterId.readClusterIdZNode=
(ZKClusterId.java:75)
at org.apache.hadoop.hbase.client.ZooKeeperRegistry.getClusterId(Zo=
oKeeperRegistry.java:86)
at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImpl=
ementation.retrieveClusterId(ConnectionManager.java:833)
at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImpl=
ementation.<init>(ConnectionManager.java:623)
... 32 more
Caused by: java.lang.IllegalArgumentException: java.net.UnknownHostExceptio=
n: svrhdfscluster
at org.apache.hadoop.security.SecurityUtil.buildTokenService(Securi=
tyUtil.java:373)
at org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNode=
Proxies.java:258)
at org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxi=
es.java:153)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:602)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:547) =
at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFi=
leSystem.java:139)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java=
:2591)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:89)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.jav=
a:2625)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2607)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:368)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296)
at org.apache.hadoop.hbase.util.DynamicClassLoader.<init>(DynamicCl=
assLoader.java:104)
at org.apache.hadoop.hbase.protobuf.ProtobufUtil.<clinit>(ProtobufU=
til.java:229)
... 37 more
Caused by: java.net.UnknownHostException: svrhdfscluster
... 51 more
sqlline version 1.1.8
0: jdbc:phoenix:svrzj001,svrzj002,svrzj003:21>

Thanks,


9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
4 years, 31 weeks ago 0|i2j4iv:
ZooKeeper ZOOKEEPER-2253

C asserts ordering of ping requests, while Java client does not

Bug Resolved Major Fixed Chris Chen Chris Chen Chris Chen 19/Aug/15 17:58   02/Mar/16 20:29 26/Sep/15 16:30 3.5.0   c client   0 4   Affects C clients from 3.3 to trunk.

The Java client does not enforce ordering on ping requests. It merely updates fields when a ping reply is received and schedules a new ping request when necessary.

The C client actually enqueues the void response in the completion data structure and pulls it off when it gets a response.

This sounds like an implementation detail (and it is, sort of), but if a future server were to, say, send unsolicited ping replies to a client to assert liveness, it would work fine against a Java client but would cause a C client to fail the assertion in zookeeper_process, "assert(cptr)", line 2912, zookeeper.c.
9223372036854775807 No Perforce job exists for this issue. 4 9223372036854775807
4 years, 25 weeks, 4 days ago 0|i2j4hb:
ZooKeeper ZOOKEEPER-2252

Random test case failure in org.apache.zookeeper.test.StaticHostProviderTest

Bug Closed Minor Fixed Timothy James Ward Mohammad Arshad Mohammad Arshad 19/Aug/15 16:09   21/Jul/16 16:18 10/Dec/15 15:44 3.5.0 3.5.2, 3.6.0     0 7   Test {{org.apache.zookeeper.test.StaticHostProviderTest.testTwoInvalidHostAddresses()}} fails randomly.

Refer bellow test ci buils:
https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/2827/testReport/

https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/2828/testReport/

https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/2830/testReport/




9223372036854775807 No Perforce job exists for this issue. 6 9223372036854775807
4 years, 15 weeks ago
Reviewed
0|i2j4cn:
ZooKeeper ZOOKEEPER-2251

Add Client side packet response timeout to avoid infinite wait.

Bug Closed Critical Fixed Mohammad Arshad nijel nijel 18/Aug/15 22:47   20/May/19 13:50 26/Jul/18 23:16 3.4.9, 3.5.2, 3.4.11 3.6.0, 3.5.5 java client   4 16 0 12600   I came across one issue related to Client side packet response timeout In my cluster many packet drops happened for some time.

One observation is the zookeeper client got hanged. As per the thread dump it is waiting for the response/ACK for the operation performed (synchronous API used here).
I am using zookeeper.serverCnxnFactory=org.apache.zookeeper.server.NIOServerCnxnFactory

Since only few packets missed there is no DISCONNECTED event occurred.

Need add a "response time out" for the operations or packets.

*Comments from [~rakeshr]*
My observation about the problem:-

* Can use tools like 'Wireshark' to simulate the artificial packet loss.
* Assume there is only one packet in the 'outgoingQueue' and unfortunately the server response packet lost. Now, client will enter into infinite waiting. https://github.com/apache/zookeeper/blob/trunk/src/java/main/org/apache/zookeeper/ClientCnxn.java#L1515
* Probably we can discuss more about this problem and possible solutions(add packet ACK timeout or another better approach) in the jira.
100% 100% 12600 0 fault, pull-request-available 9223372036854775807 No Perforce job exists for this issue. 4 9223372036854775807
1 year, 33 weeks, 6 days ago 0|i2j31z:
ZooKeeper ZOOKEEPER-2250

Support client connections using a SOCKS proxy

New Feature Open Major Unresolved Unassigned David Phillips David Phillips 17/Aug/15 19:50   07/Feb/17 11:19       java client   3 5   Connecting to ZooKeeper via a SOCKS proxy is often useful for debugging systems over an SSH dynamic port forward.

It is possible to do this today with some hacking by setting "zookeeper.clientCnxnSocket", but that is difficult because ClientCnxnSocket is package-private and is quite low-level.
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 6 weeks, 2 days ago 0|i2j153:
ZooKeeper ZOOKEEPER-2249

CRC check failed when preAllocSize smaller than node data

Bug Resolved Major Fixed Abraham Fine Benjamin Jaton Benjamin Jaton 17/Aug/15 12:24   08/Mar/18 12:40 18/Jan/18 18:48 3.5.3, 3.4.11, 3.6.0 3.5.4, 3.6.0, 3.4.12     0 5   Unexpected exception, exiting abnormally
java.io.IOException: CRC check failed
org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.next(FileTxnLog.java:612)
org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:157)
org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:223)
org.apache.zookeeper.server.ZooKeeperServer.loadData(ZooKeeperServer.java:272)
org.apache.zookeeper.server.ZooKeeperServer.startdata(ZooKeeperServer.java:399)

To reproduce, set the preAllocSize to 8MB, the jute.maxbuffer to 20MB and try saving a 15MB node several times.
In my case the erroneous CRC appears after the second save. I use the LogFormatter class to detect it.
I suspect that the CRC error happens when the new transaction log is created, the code probably expects to have enough room to save the transaction when creating a new file, but it's too small.
server 9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
2 years, 9 weeks ago 0|i2j0ev:
ZooKeeper ZOOKEEPER-2248

log.zxid file is very large size. It's likely to lead to zk disk space is not enough.

Improvement Open Major Unresolved Unassigned Yongcheng Liu Yongcheng Liu 15/Aug/15 03:54   15/Aug/15 17:20           0 3   1. about logCount(the count of log entries)
2. it's local variable in SyncRequestProcessor, it will be reset 0, when SyncRequestProcessor thread exit for LOOKING. SyncRequestProcessor thread will exit but zk process is ok, when follower followWithLeader and connecting is broken.
3. it will lead to log.zxid file keep expanding 64M
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
4 years, 31 weeks, 5 days ago 0|i2iyj3:
ZooKeeper ZOOKEEPER-2247

Zookeeper service becomes unavailable when leader fails to write transaction log

Bug Closed Critical Fixed Rakesh Radhakrishnan Mohammad Arshad Mohammad Arshad 14/Aug/15 09:13   30/Jan/19 09:31 13/Aug/16 09:59 3.5.0 3.4.9, 3.5.3, 3.6.0     0 12 0 600   Zookeeper service becomes unavailable when leader fails to write transaction log. Bellow are the exceptions
{code}
2015-08-14 15:41:18,556 [myid:100] - ERROR [SyncThread:100:ZooKeeperCriticalThread@48] - Severe unrecoverable error, from thread : SyncThread:100
java.io.IOException: Input/output error
at sun.nio.ch.FileDispatcherImpl.force0(Native Method)
at sun.nio.ch.FileDispatcherImpl.force(FileDispatcherImpl.java:76)
at sun.nio.ch.FileChannelImpl.force(FileChannelImpl.java:376)
at org.apache.zookeeper.server.persistence.FileTxnLog.commit(FileTxnLog.java:331)
at org.apache.zookeeper.server.persistence.FileTxnSnapLog.commit(FileTxnSnapLog.java:380)
at org.apache.zookeeper.server.ZKDatabase.commit(ZKDatabase.java:563)
at org.apache.zookeeper.server.SyncRequestProcessor.flush(SyncRequestProcessor.java:178)
at org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:113)
2015-08-14 15:41:18,559 [myid:100] - INFO [SyncThread:100:ZooKeeperServer$ZooKeeperServerListenerImpl@500] - Thread SyncThread:100 exits, error code 1
2015-08-14 15:41:18,559 [myid:100] - INFO [SyncThread:100:ZooKeeperServer@523] - shutting down
2015-08-14 15:41:18,560 [myid:100] - INFO [SyncThread:100:SessionTrackerImpl@232] - Shutting down
2015-08-14 15:41:18,560 [myid:100] - INFO [SyncThread:100:LeaderRequestProcessor@77] - Shutting down
2015-08-14 15:41:18,560 [myid:100] - INFO [SyncThread:100:PrepRequestProcessor@1035] - Shutting down
2015-08-14 15:41:18,560 [myid:100] - INFO [SyncThread:100:ProposalRequestProcessor@88] - Shutting down
2015-08-14 15:41:18,561 [myid:100] - INFO [SyncThread:100:CommitProcessor@356] - Shutting down
2015-08-14 15:41:18,561 [myid:100] - INFO [CommitProcessor:100:CommitProcessor@191] - CommitProcessor exited loop!
2015-08-14 15:41:18,562 [myid:100] - INFO [SyncThread:100:Leader$ToBeAppliedRequestProcessor@915] - Shutting down
2015-08-14 15:41:18,562 [myid:100] - INFO [SyncThread:100:FinalRequestProcessor@646] - shutdown of request processor complete
2015-08-14 15:41:18,562 [myid:100] - INFO [SyncThread:100:SyncRequestProcessor@191] - Shutting down
2015-08-14 15:41:18,563 [myid:100] - INFO [ProcessThread(sid:100 cport:-1)::PrepRequestProcessor@159] - PrepRequestProcessor exited loop!
{code}
After this exception Leader server still remains leader. After this non recoverable exception the leader should go down and let other followers become leader.
100% 100% 600 0 pull-request-available 9223372036854775807 No Perforce job exists for this issue. 25 9223372036854775807
2 years, 36 weeks, 5 days ago 0|i2ixev:
ZooKeeper ZOOKEEPER-2246

quorum connection manager takes a long time to shut down

Bug Open Major Unresolved Michael Han Michi Mutsuzaki Michi Mutsuzaki 12/Aug/15 22:44   05/Feb/20 07:16     3.7.0, 3.5.8 quorum   0 8   Receive worker can take a long time to shut down because the socket timeout is set to zero: http://s.apache.org/TfI

There was a discussion on the mailing list a while back: http://s.apache.org/cYG
9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
4 years, 27 weeks, 1 day ago 0|i2iuy7:
ZooKeeper ZOOKEEPER-2245

SimpleSysTest test cases fails

Bug Closed Major Fixed Mohammad Arshad Mohammad Arshad Mohammad Arshad 11/Aug/15 09:48   21/Jul/16 16:18 17/Sep/15 03:14 3.5.0 3.4.7, 3.5.2, 3.6.0     0 5   When {{org.apache.zookeeper.test.system.SimpleSysTest}} is run for in-memory Zookeeper Servers, by specifying baseSysTest.fakeMachines=yes, it fails. Its displays following errors
1:{code}
java.io.IOException: org.apache.zookeeper.server.quorum.QuorumPeerConfig$ConfigException: Address unresolved: 127.0.0.1:participant
at org.apache.zookeeper.server.quorum.Leader.lead(Leader.java:474)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1077)
Caused by: org.apache.zookeeper.server.quorum.QuorumPeerConfig$ConfigException: Address unresolved: 127.0.0.1:participant
at org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer.<init>(QuorumPeer.java:221)
{code}
2:
{code}
java.lang.NullPointerException
at org.apache.zookeeper.test.system.BaseSysTest.tearDown(BaseSysTest.java:66)
{code}
9223372036854775807 No Perforce job exists for this issue. 5 9223372036854775807
4 years, 27 weeks ago 0|i2il73:
ZooKeeper ZOOKEEPER-2244

On Windows zookeeper fails to restart

Bug Closed Critical Fixed Mohammad Arshad Mohammad Arshad Mohammad Arshad 11/Aug/15 07:26   21/Jul/16 16:18 28/Sep/15 22:12 3.5.0 3.5.2, 3.6.0     0 7   This issue occurs in following scenario
1) configure server properties in zookeeper configuration file(zoo.cfg)
example:
{code}
server.1=localhost:43222:43225:participant;0.0.0.0:43228
server.2=localhost:43223:43226:participant;0.0.0.0:43229
server.3=localhost:43224:43227:participant;0.0.0.0:43230
{code}
2) start the servers on windows. All the servers started successfully
3) stop any of the server
4) try to start the stopped server. It fails with following error
{code}
org.apache.zookeeper.server.quorum.QuorumPeerConfig$ConfigException: Error processing D:SystemTestCasesZKServer1confzoo.cfg.dynamic.100000000
{code}





9223372036854775807 No Perforce job exists for this issue. 8 9223372036854775807
4 years, 24 weeks, 3 days ago 0|i2il07:
ZooKeeper ZOOKEEPER-2243

Supported platforms is completely out of date

Bug Closed Major Fixed Chris Nauroth Ivan Kelly Ivan Kelly 11/Aug/15 04:16   21/Jul/16 16:18 08/Feb/16 16:33   3.4.9, 3.5.2, 3.6.0     0 8   http://zookeeper.apache.org/doc/r3.4.6/zookeeperAdmin.html#sc_supportedPlatforms

It refers to Solaris as Sun Solaris so it's at least 5 years out of date.

We should "support" the platforms that we are running zookeeper on regularly, so I suggest paring it down to linux and windows (mac os doesn't really count because people don't run it on servers anymore). Everything else should be "may work, not supported, but will fix obvious bugs".
9223372036854775807 No Perforce job exists for this issue. 2 9223372036854775807
4 years, 6 weeks, 3 days ago
Reviewed
0|i2ikqn:
ZooKeeper ZOOKEEPER-2242

ZooKeeper OSGi bundle is missing package import for org.ietf.jgss.

Bug Resolved Major Duplicate Unassigned Chris Nauroth Chris Nauroth 06/Aug/15 13:41   21/Aug/15 12:27 21/Aug/15 12:27     build   0 1   The ZooKeeper build injects OSGi headers into the manifest, but the {{Import-Package}} header does not include {{org.ietf.jgss}}, which is used by the ZooKeeper code. For applications using ZooKeeper inside an OSGi container, this can cause {{ClassNotFoundException}} unless the application adds the missing import to its own OSGi bundle. 9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
4 years, 30 weeks, 6 days ago 0|i2ifi7:
ZooKeeper ZOOKEEPER-2241

In Login.java, a tgt with a small expiry time will break the code

Bug Open Minor Unresolved Unassigned Reza Farivar Reza Farivar 05/Aug/15 16:18   05/Aug/15 16:18           0 1   In the Login.java code, if a TGT with a small expiration date (e.g. 5 minutes) is passed in, the refresh date is set at a value less than the MIN_TIME_BEFORE_RELOGIN, which is a minute by default. As a result, the condition in line 153 evaluates to true, setting nextRefresh to now. Then right after, in line 176, it checks the nextRefresh againt now, and will jump to line 186 and just exit (without throwing an exception), exiting the refresh thread.

https://github.com/apache/zookeeper/blob/trunk/src/java/main/org/apache/zookeeper/Login.java#L186

Possible Solution: changing line 176 to
if (now <= nextRefresh)

9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
4 years, 33 weeks, 1 day ago 0|i2idvj:
ZooKeeper ZOOKEEPER-2240

Make the three-node minimum more explicit in documentation and on website

Improvement Closed Trivial Fixed Mohammad Arshad Shawn Heisey Shawn Heisey 05/Aug/15 00:09   21/Jul/16 16:18 20/Mar/16 00:31   3.4.9, 3.5.2, 3.6.0 documentation   0 6   One of the most important parts of a production zookeeper deployment is the three-node minimum requirement for fault tolerance ... but when I glance at the website and the documentation, this requirement is difficult to actually find.

It is buried deep in the admin documentation, in a sentence that says "Thus, a deployment that consists of three machines can handle one failure, and a deployment of five machines can handle two failures." Other parts of the documentation hint at it, but nothing that I've seen comes out and explicitly says it.

Ideally, documentation about this requirement would be in a location where it can be easily pinpointed with a targeted URL, so I can point to ZK documentation with a link and clearly tell SolrCloud users that this is a real requirement.

If someone can point me to version control locations where I can check out or clone the docs and the website, I'm happy to attempt a patch.
9223372036854775807 No Perforce job exists for this issue. 5 9223372036854775807
4 years, 4 days ago
Reviewed
0|i2ich3:
ZooKeeper ZOOKEEPER-2239

JMX State from LocalPeerBean incorrect

Bug Closed Major Fixed Kevin Lee Kevin Lee Kevin Lee 04/Aug/15 12:33   21/Jul/16 16:18 26/Oct/15 03:15 3.4.6, 3.5.0, 3.5.1 3.4.7, 3.5.2, 3.6.0 jmx   0 6   All The "State" property of LocalPeerBean in package org.apache.zookeeper.server.quorum is returning the incorrect value. It is performing peer.getState() which is calling the getState() method on java.lang.Thread instead of getting the server state from org.apache.zookeeper.server.quorum.QuorumPeer. The Javadoc within LocalPeerMXBean.java states that it should be returning the server state as well. The fix is to call peer.getServerState() in the getState() method of LocalPeerBean instead of peer.getState().toString(). This will return the states defined in QuorumStats.Provider (unknown, leaderelection, leading, following, and observing). This issue prevents JMX monitoring of the Zookeeper server state. 9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
4 years, 21 weeks, 3 days ago 0|i2ibkv:
ZooKeeper ZOOKEEPER-2238

Support limiting the maximum number of connections/clients to a zookeeper server.

Improvement Resolved Major Fixed Sujith Simon nijel nijel 04/Aug/15 01:08   13/Nov/19 23:54 13/Nov/19 21:55   3.6.0     2 10 0 14400   Currently zookeeper have the feature of limiting the maximum number of connection/client per IP or Host (maxClientCnxns).

But to safe guard zookeeper server from DoS attack due to many clients from different IPs, it is better to have a limit of total number of connections/clients to a a single member of the ZooKeeper ensemble as well.

So the idea is to introduce a new configuration to limit the maximum number of total connections/clients.

Please share your thoughts.
100% 100% 14400 0 pull-request-available 9223372036854775807 No Perforce job exists for this issue. 7 9223372036854775807
18 weeks ago 0|i2iasv:
ZooKeeper ZOOKEEPER-2237

ZOOKEEPER-1572 Port async multi to 3.4 branch

Sub-task Resolved Major Fixed Ivan Kelly Ivan Kelly Ivan Kelly 30/Jul/15 04:26   31/Jul/15 23:46 31/Jul/15 23:46   3.4.7 java client   0 3   Async multi is available in 3.5 branch, but this is currently alpha, and doesn't look like it'll be GA in the next 6 months.

I've run into a few cases where async multi would be really useful, but we want to stick with a GA client. Thus, I'm going to backport Sijie's patch to 3.4.
9223372036854775807 No Perforce job exists for this issue. 2 9223372036854775807
4 years, 33 weeks, 5 days ago 0|i2i42v:
ZooKeeper ZOOKEEPER-2236

Zookeeper truncates file to 0bytes

Bug Open Major Unresolved Unassigned Bharat Singh Bharat Singh 23/Jul/15 11:46   12/Dec/16 03:36   3.4.6   contrib-zkfuse, server, tests   0 3   Ubuntu14.04, Standalone Zookeeper server, Zkfuse I am facing a rename issue with Zkfuse.
I am trying to test file atomic updates. After some iterations the file size becomes 0.
This is easily reproducible, just running the below script for ~5mins.

Setup:
zookeeper-3.4.6 with Zkfuse mounted, size of testFile = 1k

while [ 1 ]
do
cp /root/testFile /mnt/zk/testFile.tmp
mv /mnt/zk/testFile.tmp /mnt/zk/testFile
ls -larth /mnt/zk/
sleep 1
done

Zkfuse debug logs doesn't show any suspicious activity. Looks like zookeeper/zkfuse RENAME is not atomic.

But code browsing and log messages show that update have issues:
1) update is not able to pull data from zookeeper due to the _refCnt > 1,
so rename get an empty ZkfuseFile object.
2) I also hit an assert in update,
assert(newFile == false || _isOnlyRegOpen());

Now I have suspicion on the refcount logic. Have any one faced similar issues or have used Zkfuse in production environment.

9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
Important
4 years, 35 weeks ago 0|i2hujj:
ZooKeeper ZOOKEEPER-2235

License update

Bug Closed Blocker Fixed Flavio Paiva Junqueira Flavio Paiva Junqueira Flavio Paiva Junqueira 17/Jul/15 12:30   21/Jul/16 16:18 03/May/16 14:30 3.4.6, 3.5.0 3.4.7, 3.5.1, 3.5.2, 3.6.0     0 5   Updating license files and notice.txt as needed. Here is a list of the jars we are currently bundling with the release artifact with the corresponding license:

# commons-cli-1.2.jar -- ASF
# javacc.jar -- BSD license
# jline-2.11.jar -- BSD license
# servlet-api-2.5-20081211.jar - CDDL
# jackson-core-asl-1.9.11.jar -- ALv2
# jetty-6.1.26.jar -- ALv2
# log4j-1.2.16.jar -- ALv2
# jackson-mapper-asl-1.9.11.jar -- ALv2
# jetty-util-6.1.26.jar -- ALv2
# netty-3.7.0.Final.jar -- ALv2
# slf4j-log4j12-1.7.5.jar -- MIT
9223372036854775807 No Perforce job exists for this issue. 9 9223372036854775807
3 years, 46 weeks, 2 days ago
Reviewed
0|i2he5z:
ZooKeeper ZOOKEEPER-2234

Snapshot serialization race condition can lead to partial transaction and inoperable data node

Bug Open Minor Unresolved Unassigned Adam Milne-Smith Adam Milne-Smith 16/Jul/15 06:30   16/Jul/15 06:30   3.4.6       0 2   This issue can be reproduced by creating a node with a new ACL during data tree serialization after ACL cache serialization. When restoring from this snapshot without the tranlog, the state will include a node with no corresponding ACL in the ACL cache. This node will then be impossible to operate on as it will cause a MarshallingError.

If the tranlog is played over a server in this erroneous state, it does appear to correct itself. This bug means that to reliably restore from a snapshot, you must also have backed up the subsequent tranlog covering at least the transactions that were partially written to the snapshot.

Issue first described here:
http://mail-archives.apache.org/mod_mbox/zookeeper-user/201507.mbox/%3C0LzCmv-1YtgSd0Dqb-014Qqf@mrelayeu.kundenserver.de%3E

It also appears possible for a snapshot to be missing a session yet contain an ephemeral node created by that session; fortunately ZooKeeperServer.loadData() should clean these ephemerals up.
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
4 years, 36 weeks ago 0|i2hbw7:
ZooKeeper ZOOKEEPER-2233

Invalid description in the comment of LearnerHandler.syncFollower()

Improvement Open Trivial Unresolved Hitoshi Mitake Hitoshi Mitake Hitoshi Mitake 14/Jul/15 02:51   14/Jul/15 14:05           0 3   LearnerHandler.syncFollower() has a comment like below:

When leader election is completed, the leader will set its
lastProcessedZxid to be (epoch < 32). There will be no txn associated
with this zxid.

However, IIUC, the expression "epoch < 32" (comparison) should be "epoch << 32" (bitshift).

Of course the error is very trivial but it was a little bit confusing for me, so I'd like to fix it.
9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
Patch
4 years, 36 weeks, 2 days ago 0|i2h7yn:
ZooKeeper ZOOKEEPER-2232

zkperl is out of sync with CPAN releases

Improvement Open Minor Unresolved Unassigned Mark Flickinger Mark Flickinger 11/Jul/15 22:16   10/Mar/16 18:42       contrib-bindings   0 2   First some background here:
Around 6 months ago I had some bug fixes I wanted patched in zkperl, which I had been using through [CPAN|https://metacpan.org/pod/Net::ZooKeeper]. I volunteered to take over maintenance of the code, and Chris Darroch set me up with permissions to push new releases to CPAN. At the time I was unaware that Net::ZooKeeper was bundled with the rest of the ZooKeeper code, and hadn't reached out to any other ZooKeeper devs.

Since that time there have been a few CPAN releases, which included some feature requests from users([#1|https://github.com/mark-5/p5-net-zookeeper/pull/1], [#2|https://github.com/mark-5/p5-net-zookeeper/pull/2]), bug fixes([#3|https://github.com/mark-5/p5-net-zookeeper/pull/3], [#5|https://github.com/mark-5/p5-net-zookeeper/issues/5]) and test failure fixes.

I'd love for zkperl to be kept in sync with CPAN releases, and to have other ZooKeeper devs approve feature requests from users. Is there any preferred process for keeping these distributions in sync?

I don't currently have any specific ideas. I mainly wanted to start the conversation here, with other devs.
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
4 years, 2 weeks ago 0|i2h567:
ZooKeeper ZOOKEEPER-2231

ServerSocket opened by ZooKeeperServer cannot use SO_REUSEADDR under Linux

Bug Open Major Unresolved Unassigned Gábor Lipták Gábor Lipták 03/Jul/15 03:55   10/Jul/15 11:59   3.4.6   server   0 3   $ uname -a
Linux 3.19.0-20-generic #20-Ubuntu SMP Fri May 29 10:10:47 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

$ java -version
java version "1.7.0_60"
Java(TM) SE Runtime Environment (build 1.7.0_60-b19)
Java HotSpot(TM) 64-Bit Server VM (build 24.60-b09, mixed mode)
I think reporting [this stackoverflow question|http://stackoverflow.com/q/31163513/337621] to the ZooKeeper team is important.

org.apache.zookeeper.server.NIOServerCnxnFactory.configure(InetSocketAddress, int) has the following code:
{code:java}
@Override
public void configure(InetSocketAddress addr, int maxcc) throws IOException {
configureSaslLogin();

thread = new Thread(this, "NIOServerCxn.Factory:" + addr);
thread.setDaemon(true);
maxClientCnxns = maxcc;
this.ss = ServerSocketChannel.open();
ss.socket().setReuseAddress(true);
LOG.info("binding to port " + addr);
ss.socket().bind(addr);
ss.configureBlocking(false);
ss.register(selector, SelectionKey.OP_ACCEPT);
}
{code}

So the intention is to use SO_REUSEADDR. This does not work under linux (at least with the java version I use). The reason is that sun.nio.ch.ServerSocketChannelImpl.setOption(SocketOption<T>, T) used by ZooKeeper has this code:

{code:java}
public <T> ServerSocketChannel setOption(SocketOption<T> paramSocketOption, T paramT) throws IOException
{
if (paramSocketOption == null)
throw new NullPointerException();
if (!(supportedOptions().contains(paramSocketOption)))
throw new UnsupportedOperationException("'" + paramSocketOption + "' not supported");
synchronized (this.stateLock) {
if (!(isOpen()))
throw new ClosedChannelException();
if ((paramSocketOption == StandardSocketOptions.SO_REUSEADDR) && (Net.useExclusiveBind()))
{
this.isReuseAddress = ((Boolean)paramT).booleanValue();
}
else {
Net.setSocketOption(this.fd, Net.UNSPEC, paramSocketOption, paramT);
}
return this;
}
}
{code}

"Net.useExclusiveBind()" seems to give back always false under linux no matter what value is set for [sun.net.useExclusiveBind|http://www.oracle.com/technetwork/java/javase/7u25-relnotes-1955741.html#napi-win] environment entry.

If someone wants to stop and start an embedded ZooKeeper server, it can result in BindExceptions. If there would be some workaround under Linux, it would be really good.

Also under windows the sun.net.useExclusiveBind env entry seems to be important to have the SO_REUSEADDR option. Maybe it would worth to document this network setting.

I have a [test code|http://pastebin.com/Hhyfiz3Y] which can reproduce the BindException under Linux.
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
4 years, 36 weeks, 6 days ago 0|i2gtq7:
ZooKeeper ZOOKEEPER-2230

Connections fo ZooKeeper server becomes slow over time with native GSSAPI

Bug Patch Available Major Unresolved Enis Soztutar Deepesh Reja Deepesh Reja 02/Jul/15 02:13   02/Jan/20 05:45   3.4.6, 3.4.7, 3.4.8, 3.5.0 3.4.6, 3.4.7, 3.4.8, 3.5.2 server   1 13 0 1800   OS: RHEL6
Java: 1.8.0_40
Configuration:

java.env:
{noformat}
SERVER_JVMFLAGS="$SERVER_JVMFLAGS -Xmx5120m"
SERVER_JVMFLAGS="$SERVER_JVMFLAGS -Djava.security.auth.login.config=/local/apps/zookeeper-test1/conf/jaas-server.conf"
SERVER_JVMFLAGS="$SERVER_JVMFLAGS -Dsun.security.jgss.native=true"
{noformat}

jaas-server.conf:
{noformat}
Server {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
isInitiator=false
principal="zookeeper/<hostname>@<REALM>";
};
{noformat}

Process environment:
{noformat}
KRB5_KTNAME=/local/apps/zookeeper-test1/conf/keytab
ZOO_LOG_DIR=/local/apps/zookeeper-test1/log
ZOOCFGDIR=/local/apps/zookeeper-test1/conf
{noformat}
ZooKeeper server becomes slow over time when native GSSAPI is used. The connection to the server starts taking upto 10 seconds.
This is happening with ZooKeeper-3.4.6 and is fairly reproducible.

Debug logs:
{noformat}
2015-07-02 00:58:49,318 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:42405:NIOServerCnxnFactory@197] - Accepted socket connection from /<client_ip>:47942
2015-07-02 00:58:49,318 [myid:] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:42405:ZooKeeperSaslServer@78] - serviceHostname is '<zookeeper-server>'
2015-07-02 00:58:49,318 [myid:] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:42405:ZooKeeperSaslServer@79] - servicePrincipalName is 'zookeeper'
2015-07-02 00:58:49,318 [myid:] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:42405:ZooKeeperSaslServer@80] - SASL mechanism(mech) is 'GSSAPI'
2015-07-02 00:58:49,324 [myid:] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:42405:ZooKeeperSaslServer@106] - Added private credential to subject: [GSSCredential:
zookeeper@<zookeeper-server> 1.2.840.113554.1.2.2 Accept [class sun.security.jgss.wrapper.GSSCredElement]]
2015-07-02 00:58:59,441 [myid:] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:42405:ZooKeeperServer@810] - Session establishment request from client /<client_ip>:47942 client's lastZxid is 0x0
2015-07-02 00:58:59,441 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:42405:ZooKeeperServer@868] - Client attempting to establish new session at /<client_ip>:47942
2015-07-02 00:58:59,448 [myid:] - DEBUG [SyncThread:0:FinalRequestProcessor@88] - Processing request:: sessionid:0x14e486028785c81 type:createSession cxid:0x0 zxid:0x110e79 txntype:-10 reqpath:n/a
2015-07-02 00:58:59,448 [myid:] - DEBUG [SyncThread:0:FinalRequestProcessor@160] - sessionid:0x14e486028785c81 type:createSession cxid:0x0 zxid:0x110e79 txntype:-10 reqpath:n/a
2015-07-02 00:58:59,448 [myid:] - INFO [SyncThread:0:ZooKeeperServer@617] - Established session 0x14e486028785c81 with negotiated timeout 10000 for client /<client_ip>:47942
2015-07-02 00:58:59,452 [myid:] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:42405:ZooKeeperServer@949] - Responding to client SASL token.
2015-07-02 00:58:59,452 [myid:] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:42405:ZooKeeperServer@953] - Size of client SASL token: 706
2015-07-02 00:58:59,460 [myid:] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:42405:ZooKeeperServer@984] - Size of server SASL response: 161
2015-07-02 00:58:59,462 [myid:] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:42405:ZooKeeperServer@949] - Responding to client SASL token.
2015-07-02 00:58:59,462 [myid:] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:42405:ZooKeeperServer@953] - Size of client SASL token: 0
2015-07-02 00:58:59,462 [myid:] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:42405:ZooKeeperServer@984] - Size of server SASL response: 32
2015-07-02 00:58:59,463 [myid:] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:42405:ZooKeeperServer@949] - Responding to client SASL token.
2015-07-02 00:58:59,463 [myid:] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:42405:ZooKeeperServer@953] - Size of client SASL token: 32
2015-07-02 00:58:59,464 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:42405:SaslServerCallbackHandler@118] - Successfully authenticated client: authenticationID=<user_principal>; authorizationID=<user_principal>.
2015-07-02 00:58:59,464 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:42405:ZooKeeperServer@964] - adding SASL authorization for authorizationID: <user_principal>
2015-07-02 00:58:59,465 [myid:] - INFO [ProcessThread(sid:0 cport:-1)::PrepRequestProcessor@494] - Processed session termination for sessionid: 0x14e486028785c81
2015-07-02 00:58:59,467 [myid:] - DEBUG [SyncThread:0:FinalRequestProcessor@88] - Processing request:: sessionid:0x14e486028785c81 type:closeSession cxid:0x1 zxid:0x110e7a txntype:-11 reqpath:n/a
2015-07-02 00:58:59,467 [myid:] - DEBUG [SyncThread:0:FinalRequestProcessor@160] - sessionid:0x14e486028785c81 type:closeSession cxid:0x1 zxid:0x110e7a txntype:-11 reqpath:n/a
2015-07-02 00:58:59,467 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:42405:NIOServerCnxn@1007] - Closed socket connection for client /<client_ip>:47942 which had sessionid 0x14e486028785c81
{noformat}

If you see, after adding the credentials to privateCredential set, it takes roughly 10 seconds to reach to session establishment request. From the code it looks like Subject.doAs() is taking a lot of time.

I connected it to jdb while it was waiting and got following stacktrace:
{noformat}
NIOServerCxn.Factory:0.0.0.0/0.0.0.0:58909:
[1] java.util.HashMap$TreeNode.find (HashMap.java:1,865)
[2] java.util.HashMap$TreeNode.find (HashMap.java:1,861)
[3] java.util.HashMap$TreeNode.find (HashMap.java:1,861)
[4] java.util.HashMap$TreeNode.find (HashMap.java:1,861)
[5] java.util.HashMap$TreeNode.find (HashMap.java:1,861)
[6] java.util.HashMap$TreeNode.find (HashMap.java:1,861)
[7] java.util.HashMap$TreeNode.find (HashMap.java:1,861)
[8] java.util.HashMap$TreeNode.putTreeVal (HashMap.java:1,981)
[9] java.util.HashMap.putVal (HashMap.java:637)
[10] java.util.HashMap.put (HashMap.java:611)
[11] java.util.HashSet.add (HashSet.java:219)
[12] javax.security.auth.Subject$ClassSet.populateSet (Subject.java:1,418)
[13] javax.security.auth.Subject$ClassSet.<init> (Subject.java:1,372)
[14] javax.security.auth.Subject.getPrivateCredentials (Subject.java:767)
[15] sun.security.jgss.GSSUtil$1.run (GSSUtil.java:340)
[16] sun.security.jgss.GSSUtil$1.run (GSSUtil.java:332)
[17] java.security.AccessController.doPrivileged (native method)
[18] sun.security.jgss.GSSUtil.searchSubject (GSSUtil.java:332)
[19] sun.security.jgss.wrapper.NativeGSSFactory.getCredFromSubject (NativeGSSFactory.java:53)
[20] sun.security.jgss.wrapper.NativeGSSFactory.getCredentialElement (NativeGSSFactory.java:116)
[21] sun.security.jgss.GSSManagerImpl.getCredentialElement (GSSManagerImpl.java:193)
[22] sun.security.jgss.GSSCredentialImpl.add (GSSCredentialImpl.java:427)
[23] sun.security.jgss.GSSCredentialImpl.<init> (GSSCredentialImpl.java:62)
[24] sun.security.jgss.GSSManagerImpl.createCredential (GSSManagerImpl.java:154)
[25] com.sun.security.sasl.gsskerb.GssKrb5Server.<init> (GssKrb5Server.java:108)
[26] com.sun.security.sasl.gsskerb.FactoryImpl.createSaslServer (FactoryImpl.java:85)
[27] javax.security.sasl.Sasl.createSaslServer (Sasl.java:524)
[28] org.apache.zookeeper.server.ZooKeeperSaslServer$1.run (ZooKeeperSaslServer.java:118)
[29] org.apache.zookeeper.server.ZooKeeperSaslServer$1.run (ZooKeeperSaslServer.java:114)
[30] java.security.AccessController.doPrivileged (native method)
[31] javax.security.auth.Subject.doAs (Subject.java:422)
[32] org.apache.zookeeper.server.ZooKeeperSaslServer.createSaslServer (ZooKeeperSaslServer.java:114)
[33] org.apache.zookeeper.server.ZooKeeperSaslServer.<init> (ZooKeeperSaslServer.java:48)
[34] org.apache.zookeeper.server.NIOServerCnxn.<init> (NIOServerCnxn.java:100)
[35] org.apache.zookeeper.server.NIOServerCnxnFactory.createConnection (NIOServerCnxnFactory.java:161)
[36] org.apache.zookeeper.server.NIOServerCnxnFactory.run (NIOServerCnxnFactory.java:202)
[37] java.lang.Thread.run (Thread.java:745)
{noformat}

This doesn't happen when we use JGSS, I think because adding credential to privateCredential set for every connection is causing Subject.doAS() to take much longer time.
100% 100% 1800 0 patch, pull-request-available 9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
11 weeks ago Fix slowness in connections when setup with native GSSAPI. kerberos, native-gssapi 0|i2grxb:
ZooKeeper ZOOKEEPER-2229

Several four-letter words are undocumented.

Bug Closed Major Fixed Chris Nauroth Chris Nauroth Chris Nauroth 01/Jul/15 20:18   21/Jul/16 16:18 10/Dec/15 01:16   3.4.8, 3.5.2, 3.6.0 documentation   0 5   The {{isro}}, {{gtmk}} and {{stmk}} commands are not covered in the four-letter word documentation. 9223372036854775807 No Perforce job exists for this issue. 2 9223372036854775807
4 years, 15 weeks ago 0|i2grkn:
ZooKeeper ZOOKEEPER-2228

WorkerReceiver's main loop (in FastLeaderElection's) should break loop upon restart

Bug Resolved Major Not A Problem Unassigned Raúl Gutiérrez Segalés Raúl Gutiérrez Segalés 01/Jul/15 19:46   02/Jul/15 21:40 02/Jul/15 21:40     server   0 3   It seems like in FastLeaderElection#Messenger#WorkerReceiver the main loop should be left immediately after this path \[0\] is taken:

{code}
if (!rqv.equals(curQV)) {
LOG.info("restarting leader election");
self.shuttingDownLE = true;
self.getElectionAlg().shutdown();
}
{code}

Instead, it keeps going which means the received message would still be applied and a new message might be send out. Should there be a break statement right after self.getElectionAlg().shutdown()?

Any ideas [~shralex]?

\[0\]: https://github.com/apache/zookeeper/blob/trunk/src/java/main/org/apache/zookeeper/server/quorum/FastLeaderElection.java#L300
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
4 years, 37 weeks, 6 days ago 0|i2griv:
ZooKeeper ZOOKEEPER-2227

stmk four-letter word fails execution at server while reading trace mask argument.

Bug Closed Major Fixed Chris Nauroth Chris Nauroth Chris Nauroth 01/Jul/15 15:22   21/Jul/16 16:18 08/Nov/15 17:31 3.3.0 3.4.7, 3.5.2, 3.6.0 server   0 7   When the server handles the {{stmk}} four-letter word, it attempts to read an 8-byte Java {{long}} from the request as the trace mask argument. The read fails, because the destination buffer's capacity is only 4 bytes. 9223372036854775807 No Perforce job exists for this issue. 3 9223372036854775807
4 years, 19 weeks, 4 days ago 0|i2gr4n:
ZooKeeper ZOOKEEPER-2226

Mixing sequential and non-sequential can throw NodeExists for sequential nodes

Bug Open Major Unresolved Unassigned David Capwell David Capwell 01/Jul/15 14:22   23/Feb/19 03:53   3.4.5       0 2   I have the following code (in curator):

{code}
int id = extractId(client.create().creatingParentsIfNeeded().withMode(CreateMode.PERSISTENT_SEQUENTIAL).forPath(prefix,
data));
{code}

and

{code}
client.create().creatingParentsIfNeeded().withMode(CreateMode.PERSISTENT).forPath(path(id),
data);
{code}

The first part joins our cluster and gets a id from zookeeper. The
second call will create a znode that looks like a znode above.

The reason I do this is that I would like for ops to be able to define
the ids when they want and not always have to (other code will
"setData" one of the paths defined above, leaving out since thats not
having issues).

I created a test case and the error thrown was not what I was expecing: Node Exists

Here is the test:

create 4 PERSISTENT znodes with ids 1, 2, 3, 4
create 1 PERSISTENT_SEQUENTIAL znode (change id = 4, so conflicts with above)

Here is the error I saw

INFO 2015-07-01 10:46:46,349 [ProcessThread(sid:0 cport:-1):]
[PrepRequestProcessor] [line 627] Got user-level KeeperException when
processing sessionid:0x14e4aba4d490000 type:create cxid:0x25 zxid:0xe
txntype:-1 reqpath:n/a Error
Path:/test/MembershipTest/replaceFourRegisterOne/member-0000000004
Error:KeeperErrorCode = NodeExists for
/test/MembershipTest/replaceFourRegisterOne/member-0000000004

org.apache.zookeeper.KeeperException$NodeExistsException:
KeeperErrorCode = NodeExists for
/test/MembershipTest/replaceFourRegisterOne/member-
...
Caused by: org.apache.zookeeper.KeeperException$NodeExistsException:
KeeperErrorCode = NodeExists for
/test/MembershipTest/replaceFourRegisterOne/member-
at org.apache.zookeeper.KeeperException.create(KeeperException.java:119)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
at org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:688)
at org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:672)
at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:107)
at org.apache.curator.framework.imps.CreateBuilderImpl.pathInForeground(CreateBuilderImpl.java:668)
at org.apache.curator.framework.imps.CreateBuilderImpl.protectedPathInForeground(CreateBuilderImpl.java:453)
at org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:443)
at org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:44)


When using sequential nodes, its unexpected that they can fail because a node already exists.
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
1 year, 3 weeks, 5 days ago 0|i2gr1j:
ZooKeeper ZOOKEEPER-2225

modify existing 'configuration' command or add new one to return list of ZooKeeper's ensemble members

Improvement Open Major Unresolved Unassigned Grigoriy Starchenko Grigoriy Starchenko 30/Jun/15 09:56   30/Jun/15 12:52       server   0 3   Linux, AWS cloud Hi, All

We are building high-availability ZooKeeper cluster at AWS and using version 3.5.0 because it support dynamic re-configuration. Everything works but one problem: it is difficult for ZooKeeper clients to discover current ensemble list. Obvious solution is to put ZooKeeper behind AWS load balancer. Client during initialization will call ZooKeeper via load balancer to read /zookeeper/config and will be able to build connection string. We quickly discovered ZooKeeper API not working trough AWS load-balancer.

ZooKeeper, starting from 3.5.0, support AdminServer option which working just fine behind any type of load balancers. The catch is: no command avail to date to get list of hosts representing ensemble.

http://localhost:8080/commands/...

provide a lot of info but none of them returns

{code}
server.4108=10.50.4.108:2888:3888:participant;0.0.0.0:2181
server.316=10.50.3.16:2888:3888:participant;0.0.0.0:2181
server.1215=10.50.1.215:2888:3888:participant;0.0.0.0:2181
version=100000000
{code}

I think it would be very useful add new command:

http://localhost:8080/commands/dconfig

which will return current ZooKeeper dynamic comfiguration:

{code}
{
"server.4108" : "10.50.4.108:2888:3888:participant;0.0.0.0:2181",
"server.316" : "10.50.3.16:2888:3888:participant;0.0.0.0:2181",
"server.1215" : "10.50.1.215:2888:3888:participant;0.0.0.0:2181"
"version" : "100000000"
}
{code}

Thank you,
Grisha
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
4 years, 38 weeks, 2 days ago 0|i2goqv:
ZooKeeper ZOOKEEPER-2224

Four letter command hangs when network is slow

Bug Resolved Minor Fixed Mohammad Arshad Mohammad Arshad Mohammad Arshad 30/Jun/15 06:00   25/Aug/15 23:40 06/Jul/15 11:50   3.4.7, 3.5.1, 3.6.0 java client   0 9   Four letter command hangs when network is slow or network goes down in between the operation, and the application also, which calling this four letter command, hangs. 9223372036854775807 No Perforce job exists for this issue. 5 9223372036854775807
4 years, 30 weeks, 1 day ago 0|i2godz:
ZooKeeper ZOOKEEPER-2223

support method-level JUnit testcase

Improvement Resolved Minor Fixed Akihiro Suda Akihiro Suda Akihiro Suda 30/Jun/15 02:26   28/Jul/15 02:47 07/Jul/15 02:46   3.5.1, 3.6.0 tests   0 6   Currently, a user can execute class-level single test, but cannot execute method-level ones.

This patch adds a support for method-level single test so as to facilitate ease of debugging failing tests (like ZOOKEEPER-2080).

Class-level test (exists in current version)
{panel}
$ ant -Dtestcase=ReconfigRecoveryTest test-core-java
{panel}

Method-level test (proposal)
{panel}
$ ant -Dtestcase=ReconfigRecoveryTest -Dtest.method=testCurrentObserverIsParticipantInNewConfig test-core-java
{panel}
9223372036854775807 No Perforce job exists for this issue. 4 9223372036854775807
Patch
4 years, 37 weeks, 2 days ago 0|i2go0f:
ZooKeeper ZOOKEEPER-2222

Fail fast if `myid` does not exist but server.N properties are defined

Improvement Resolved Minor Invalid Unassigned Joe Halliwell Joe Halliwell 26/Jun/15 12:17   26/Jun/15 13:01 26/Jun/15 13:01 3.4.6   server   0 1   Under these circumstances the server logs a warning, but starts in standalone mode. I think it should exit. 9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
4 years, 38 weeks, 6 days ago 0|i2gjvr:
ZooKeeper ZOOKEEPER-2221

Zookeeper JettyAdminServer server should start on configured IP.

Bug Resolved Major Fixed Surendra Singh Lilhore Surendra Singh Lilhore Surendra Singh Lilhore 25/Jun/15 10:07   28/Jul/15 02:48 30/Jun/15 14:51 3.5.0 3.5.1, 3.6.0 server   0 7   Currently JettyAdminServer starting on "0.0.0.0" IP. "0.0.0.0" means "all IP addresses on the local machine". So, if your webserver machine has two ip addresses, 192.168.1.1(private) and 10.1.2.1(public), and you allow a webserver daemon like apache to listen on 0.0.0.0, it will be reachable at both of those IPs.

This is security issue. webserver should be accessible from only configured IP
9223372036854775807 No Perforce job exists for this issue. 5 9223372036854775807
4 years, 38 weeks, 1 day ago 0|i2ghsn:
ZooKeeper ZOOKEEPER-2220

Couldn't instantiate org.apache.zookeeper.ClientCnxnSocketNetty

Bug Open Major Unresolved Unassigned rupa mogali rupa mogali 24/Jun/15 13:57   10/Oct/17 01:50   3.5.0   c client   0 4   Alpha I am trying to test SSL connectivity between client and server following the instructions in the following page:
https://cwiki.apache.org/confluence/display/ZOOKEEPER/ZooKeeper+SSL+User+Guide
But, I get the following when trying to connect to server from client..
2015-06-24 12:14:36,589 [myid:] - INFO [main:ZooKeeper@709] - Initiating client connection, connectString=localhost:2282 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@f2a0b8e
Exception in thread "main" java.io.IOException: Couldn't instantiate org.apache.zookeeper.ClientCnxnSocketNetty
Can you tell me what I am doing wrong here?
Very new to Zookeeper.
Thanks!
Reply
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
4 years, 38 weeks, 3 days ago 0|i2gg3r:
ZooKeeper ZOOKEEPER-2219

ZooKeeper server should better handle SessionMovedException.

Bug Open Major Unresolved Unassigned Zhihai Xu Zhihai Xu 24/Jun/15 02:39   21/Nov/18 21:09   3.4.5       2 6   ZooKeeper server should better handle SessionMovedException.
We hit the SessionMovedException. the following is the reason for the SessionMovedException we find:
1. ZK client tried to connect to Leader L. Network was very slow, so before leader processed the request, client disconnected.
2. Client then re-connected to Follower F reusing the same session ID. It was successful.
3. The request in step 1 went into leader. Leader processed it and invalidated the connection created in step 2. But client didn't know the connection it used is invalidated.
4. Client got SessionMovedException when it used the connection invalidated by leader for any ZooKeeper operation.

The following are logs: c045dkh is the Leader, c470udy is the Follower and the sessionID is 0x14be28f50f4419d.
1. ZK client try to initiate session to Leader at 015-03-16 10:59:40,735 and timeout after 10/3 seconds.
logs from ZK client
{code}
2015-03-16 10:59:40,078 INFO org.apache.zookeeper.ClientCnxn: Client session timed out, have not heard from server in 6670ms for sessionid 0x14be28f50f4419d, closing socket connection and attempting reconnect
015-03-16 10:59:40,735 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server c045dkh/?.?.?.67:2181. Will not attempt to authenticate using SASL (unknown error)
2015-03-16 10:59:40,735 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to c045dkh/?.?.?.67:2181, initiating session
2015-03-16 10:59:44,071 INFO org.apache.zookeeper.ClientCnxn: Client session timed out, have not heard from server in 3336ms for sessionid 0x14be28f50f4419d, closing socket connection and attempting reconnect
{code}

2. ZK client initiated session to Follower successfully at 2015-03-16 10:59:44,688
logs from ZK client
{code}
2015-03-16 10:59:44,673 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server c470udy/?.?.?.65:2181. Will not attempt to authenticate using SASL (unknown error)
2015-03-16 10:59:44,673 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to c470udy/?.?.?.65:2181, initiating session
2015-03-16 10:59:44,688 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server c470udy/?.?.?.65:2181, sessionid = 0x14be28f50f4419d, negotiated timeout = 10000
{code}
logs from ZK Follower server
{code}
2015-03-16 10:59:44,673 INFO org.apache.zookeeper.server.NIOServerCnxnFactory: Accepted socket connection from /?.?.?.65:42777
2015-03-16 10:59:44,674 INFO org.apache.zookeeper.server.ZooKeeperServer: Client attempting to renew session 0x14be28f50f4419d at /?.?.?.65:42777
2015-03-16 10:59:44,674 INFO org.apache.zookeeper.server.quorum.Learner: Revalidating client: 0x14be28f50f4419d
2015-03-16 10:59:44,675 INFO org.apache.zookeeper.server.ZooKeeperServer: Established session 0x14be28f50f4419d with negotiated timeout 10000 for client /?.?.?.65:42777
{code}

3. At 2015-03-16 10:59:45,668, Leader processed the delayed request which is sent by Client at 2015-03-16 10:59:40,735, right after session was established, it close the socket connection/session because client was already disconnected due to timeout.
logs from ZK Leader server
{code}
2015-03-16 10:59:45,668 INFO org.apache.zookeeper.server.ZooKeeperServer: Client attempting to renew session 0x14be28f50f4419d at /?.?.?.65:50271
2015-03-16 10:59:45,668 INFO org.apache.zookeeper.server.ZooKeeperServer: Established session 0x14be28f50f4419d with negotiated timeout 10000 for client /?.?.?.65:50271
2015-03-16 10:59:45,670 WARN org.apache.zookeeper.server.NIOServerCnxn: Exception causing close of session 0x14be28f50f4419d due to java.io.IOException: Broken pipe
2015-03-16 10:59:45,671 INFO org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection for client /?.?.?.65:50271 which had sessionid 0x14be28f50f4419d
{code}

4. Client got SessionMovedException at 2015-03-16 10:59:45,693
logs from ZK Leader server
{code}
2015-03-16 10:59:45,693 INFO org.apache.zookeeper.server.PrepRequestProcessor: Got user-level KeeperException when processing sessionid:0x14be28f50f4419d type:multi cxid:0x86e3 zxid:0x1c002a4e53 txntype:-1 reqpath:n/a aborting remaining multi ops. Error Path:null Error:KeeperErrorCode = Session moved
2015-03-16 10:59:45,695 INFO org.apache.zookeeper.server.PrepRequestProcessor: Got user-level KeeperException when processing sessionid:0x14be28f50f4419d type:multi cxid:0x86e5 zxid:0x1c002a4e56 txntype:-1 reqpath:n/a aborting remaining multi ops. Error Path:null Error:KeeperErrorCode = Session moved
2015-03-16 10:59:45,700 INFO org.apache.zookeeper.server.PrepRequestProcessor: Got user-level KeeperException when processing sessionid:0x14be28f50f4419d type:multi cxid:0x86e7 zxid:0x1c002a4e57 txntype:-1 reqpath:n/a aborting remaining multi ops. Error Path:null Error:KeeperErrorCode = Session moved
{code}

5. At 2015-03-16 10:59:45,710, we close the session 0x14be28f50f4419d but the socket connection between ZK client and ZK Follower is closed at 2015-03-16 10:59:45,715 after session termination.
logs from ZK Leader server:
{code}
2015-03-16 10:59:45,710 INFO org.apache.zookeeper.server.PrepRequestProcessor: Processed session termination for sessionid: 0x14be28f50f4419d
{code}
logs from ZK Follower server:
{code}
2015-03-16 10:59:45,715 INFO org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection for client /?.?.?.65:42777 which had sessionid 0x14be28f50f4419d
{code}

It looks like Zk client is out-of-sync with ZK server.
My question is how ZK client can recover from this error. It looks like the ZK Client won't be disconnected from Follower until session is closed and any ZooKeeper operation will fail with SessionMovedException before session is closed.
Also since ZK Leader already closed the socket connection/session to ZK Client at step 3, why it still reject the ZooKeeper operation from client with SessionMovedException. Will it be better to endorse the session/connection between ZK client and ZK Follower? This seems like a bug to me.
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
1 year, 17 weeks ago 0|i2gf4v:
ZooKeeper ZOOKEEPER-2218

Close IO Streams in finally block

Improvement Reopened Major Unresolved Tang Xinye Tang Xinye Tang Xinye 22/Jun/15 22:32   14/Dec/19 06:08     3.7.0     0 6 0 600   The problem here is that if an exception is thrown during the read process the method will exit without closing the stream and hence without releasing the file system resources, it may run out of resources before it does run. 100% 100% 600 0 pull-request-available 9223372036854775807 No Perforce job exists for this issue. 2 9223372036854775807
1 year, 20 weeks, 6 days ago Place the close method in the finally clause, so we can ensure it always runs regardless of how the method exits. 0|i2gd9b:
ZooKeeper ZOOKEEPER-2217

event might lost before re-watch

Improvement Resolved Major Not A Problem Unassigned Caspian Caspian 18/Jun/15 23:25   26/Jun/15 23:01 26/Jun/15 23:01 3.4.5, 3.4.6   c client, java client   1 4   jdk1.7_45 on centos6.5 and ubuntu14.4 I use zk to monitor the children nodes under a path, eg: /servers.
when the client is told that children changes, I have to re-watch the path again, during the peroid, it's possible that some children down, or some up. And those events will be missed.

For now, my temporary solution is not to use getChildren(path, true...) to get children and re-watch this path, but re-watch this path first, then get the children. Thus non events can be ignored, but I don't know what will the zk server be like if there are too much clients that act like this.

How do you think of this problem? Is there any other solutions?

9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
4 years, 38 weeks, 5 days ago 0|i2g927:
ZooKeeper ZOOKEEPER-2216

Get the property hierarchy as a a whole tree

Improvement Open Minor Unresolved Unassigned Nabarun Mondal Nabarun Mondal 16/Jun/15 13:38   16/Jun/15 13:38   3.5.0   c client   0 1   I am logging this as part of a feature request.
We use Zookeeper - pretty extensively.
Thanks for putting a pretty awesome product!

This is a feature request.
As of now, there is no way to ask Zookeeper to get the whole property hierarchy as a whole tree in a single call.
We would be grateful if you guys can give this facility to get the whole property tree as a whole.

NOTE: I personally won't mind coding this, if you guys permit me.
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
4 years, 40 weeks, 2 days ago 0|i2g44f:
ZooKeeper ZOOKEEPER-2215

Four letter command don't have kerberos authentication

Bug Open Major Unresolved Unassigned Surendra Singh Lilhore Surendra Singh Lilhore 12/Jun/15 08:09   13/Jun/15 01:34           0 2   echo dump | netcat <IP> <port> 9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
4 years, 40 weeks, 5 days ago 0|i2fz07:
ZooKeeper ZOOKEEPER-2214

Findbugs warning: LearnerHandler.packetToString Dead store to local variable

Improvement Resolved Minor Fixed Hongchao Deng Hongchao Deng Hongchao Deng 11/Jun/15 11:46   13/Jun/15 06:37 12/Jun/15 17:04   3.5.1, 3.6.0     0 3   9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
4 years, 40 weeks, 5 days ago 0|i2fxrb:
ZooKeeper ZOOKEEPER-2213

Empty path in Set crashes server and prevents restart

Bug Resolved Blocker Fixed Hongchao Deng Brian Brazil Brian Brazil 10/Jun/15 11:29   29/Jun/15 13:43 11/Jun/15 14:16 3.4.5 3.4.7, 3.5.1, 3.6.0 server   0 7   See https://github.com/samuel/go-zookeeper/issues/62

I've reproduced this on 3.4.5 with the code:
c, _, _ := zk.Connect([]string{"127.0.0.1"}, time.Second)
c.Set("", []byte{}, 0)

This crashes a local zookeeper 3.4.5 server:
2015-06-10 16:21:10,862 [myid:] - ERROR [SyncThread:0:SyncRequestProcessor@151] - Severe unrecoverable error, exiting
java.lang.IllegalArgumentException: Invalid path
at org.apache.zookeeper.common.PathTrie.findMaxPrefix(PathTrie.java:259)
at org.apache.zookeeper.server.DataTree.getMaxPrefixWithQuota(DataTree.java:634)
at org.apache.zookeeper.server.DataTree.setData(DataTree.java:616)
at org.apache.zookeeper.server.DataTree.processTxn(DataTree.java:807)
at org.apache.zookeeper.server.ZKDatabase.processTxn(ZKDatabase.java:329)
at org.apache.zookeeper.server.ZooKeeperServer.processTxn(ZooKeeperServer.java:965)
at org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:116)
at org.apache.zookeeper.server.SyncRequestProcessor.flush(SyncRequestProcessor.java:167)
at org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:101)

On restart the zookeeper server crashes out:
2015-06-10 16:22:21,352 [myid:] - ERROR [main:ZooKeeperServerMain@54] - Invalid arguments, exiting abnormally
java.lang.IllegalArgumentException: Invalid path
at org.apache.zookeeper.common.PathTrie.findMaxPrefix(PathTrie.java:259)
at org.apache.zookeeper.server.DataTree.getMaxPrefixWithQuota(DataTree.java:634)
at org.apache.zookeeper.server.DataTree.setData(DataTree.java:616)
at org.apache.zookeeper.server.DataTree.processTxn(DataTree.java:807)
at org.apache.zookeeper.server.persistence.FileTxnSnapLog.processTransaction(FileTxnSnapLog.java:198)
at org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:151)
at org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:223)
at org.apache.zookeeper.server.ZooKeeperServer.loadData(ZooKeeperServer.java:250)
at org.apache.zookeeper.server.ZooKeeperServer.startdata(ZooKeeperServer.java:377)
at org.apache.zookeeper.server.NIOServerCnxnFactory.startup(NIOServerCnxnFactory.java:122)
at org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:112)
at org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:86)
at org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:52)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:116)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:78)
9223372036854775807 No Perforce job exists for this issue. 5 9223372036854775807
4 years, 38 weeks, 3 days ago 0|i2fvtr:
ZooKeeper ZOOKEEPER-2212

distributed race condition related to QV version

Bug Resolved Critical Fixed Akihiro Suda Akihiro Suda Akihiro Suda 10/Jun/15 02:45   14/Aug/15 00:12 15/Jun/15 19:08 3.5.0 3.5.1, 3.6.0 quorum   0 8   When a joiner is listed as an observer in an initial config,
the joiner should become a non-voting follower (not an observer) until reconfig is triggered. [(Link)|http://zookeeper.apache.org/doc/trunk/zookeeperReconfig.html#sc_reconfig_general]

I found a distributed race-condition situation where an observer keeps being an observer and cannot become a non-voting follower.

This race condition happens when an observer receives an UPTODATE Quorum Packet from the leader:2888/tcp *after* receiving a Notification FLE Packet of which n.config version is larger than the observer's one from leader:3888/tcp.

h4. Detail
* Problem: An observer cannot become a non-voting follower
* Cause: Cannot restart FLE
* Cause: In {{QuorumPeer.run()}}, cannot shutdown {{Observer}} [(Link)|https://github.com/apache/zookeeper/blob/98a3cabfa279833b81908d72f1c10ee9f598a045/src/java/main/org/apache/zookeeper/server/quorum/QuorumPeer.java#L1014]
* Cause: In {{QuorumPeer.run()}}, cannot return from {{Observer.observeLeader()}} [(Link)|https://github.com/apache/zookeeper/blob/98a3cabfa279833b81908d72f1c10ee9f598a045/src/java/main/org/apache/zookeeper/server/quorum/QuorumPeer.java#L1010]
* Cause: In {{Observer.observeLeader()}}, {{Learner.syncWithLeader()}} does not throw an exception of "changes proposed in reconfig" [(Link)|https://github.com/apache/zookeeper/blob/98a3cabfa279833b81908d72f1c10ee9f598a045/src/java/main/org/apache/zookeeper/server/quorum/Observer.java#L79]
* Cause: In {{switch(qp.getType()) case UPTODATE}} of {{Learner.syncWithLeader()}} [(Link)|https://github.com/apache/zookeeper/blob/98a3cabfa279833b81908d72f1c10ee9f598a045/src/java/main/org/apache/zookeeper/server/quorum/Learner.java#L492-507], {{QuorumPeer.processReconfig()}} [(Link)|https://github.com/apache/zookeeper/blob/98a3cabfa279833b81908d72f1c10ee9f598a045/src/java/main/org/apache/zookeeper/server/quorum/QuorumPeer.java#L1644]returns false with a log message like ["2 setQuorumVerifier called with known or old config 4294967296. Current version: 4294967296"|https://github.com/osrg/earthquake/blob/v0.1/example/zk-found-bug.ether/example-output/3.REPRODUCED/zk2.log]. [(Link)|https://github.com/apache/zookeeper/blob/98a3cabfa279833b81908d72f1c10ee9f598a045/src/java/main/org/apache/zookeeper/server/quorum/QuorumPeer.java#L1369]
,
* Cause: The observer have already received a Notification Packet({{n.config.version=4294967296}}) and invoked {{QuorumPeer.processReconfig()}} [(Link)|https://github.com/apache/zookeeper/blob/98a3cabfa279833b81908d72f1c10ee9f598a045/src/java/main/org/apache/zookeeper/server/quorum/FastLeaderElection.java#L291-304]

h4. How I found this bug
I found this bug using [Earthquake|http://osrg.github.io/earthquake/], our open-source dynamic model checker for real implementations of distributed systems.

Earthquakes permutes C/Java function calls, Ethernet packets, and injected fault events in various orders so as to find implementation-level bugs of the distributed system.

When Earthquake finds a bug, Earthquake automatically records [the event history|https://github.com/osrg/earthquake/blob/v0.1/example/zk-found-bug.ether/example-output/3.REPRODUCED/json] and helps the user to analyze which permutation of events triggers the bug.

I analyzed Earthquake's event histories and found that the bug is triggered when an observer receives an UPTODATE *after* receiving a specific kind of FLE packet.

h4. How to reproduce this bug
You can also easily reproduce the bug using Earthquake.
I made a Docker container [osrg/earthquake-zookeeper-2212|https://registry.hub.docker.com/u/osrg/earthquake-zookeeper-2212/] on Docker hub:
{code}
host$ sudo modprobe openvswitch
host$ docker run --privileged -t -i --rm osrg/earthquake-zookeeper-2212
guest$ ./000-prepare.sh
[INFO] Starting Earthquake Ethernet Switch
[INFO] Starting Earthquake Orchestrator
[INFO] Starting Earthquake Ethernet Inspector
[IMPORTANT] Please kill the processes (switch=1234, orchestrator=1235, and inspector=1236) after you finished all of the experiments
[IMPORTANT] Please continue to 100-run-experiment.sh..
guest$ ./100-run-experiment.sh
[IMPORTANT] THE BUG WAS REPRODUCED!
guest$ kill -9 1234 1235 1236
{code}

Note that {{--privileged}} is needed, as this container uses Docker-in-Docker.

For further information about reproducing this bug, please refer to https://github.com/osrg/earthquake/blob/v0.1/example/zk-found-bug.ether
9223372036854775807 No Perforce job exists for this issue. 3 9223372036854775807
4 years, 31 weeks, 6 days ago https://github.com/osrg/earthquake/tree/v0.1/example/zk-found-bug.ether 0|i2fv5r:
ZooKeeper ZOOKEEPER-2211

PurgeTxnLog does not correctly purge when snapshots and logs are at different locations

Bug Closed Major Fixed Mohammad Arshad Wesley Chow Wesley Chow 09/Jun/15 16:45   21/Jul/16 16:18 03/Dec/15 23:20 3.4.6, 3.5.0 3.4.8, 3.5.2, 3.6.0 scripts   0 8   Ubuntu 12.04, Java 1.7. PurgeTxnLog does not work when snapshots and transaction logs are at different file paths. The argument handling is buggy and only works when both snap and datalog dirs are given, and datalog dir contains both logs and snaps (snap is ignored). 9223372036854775807 No Perforce job exists for this issue. 5 9223372036854775807
Patch
4 years, 15 weeks, 6 days ago 0|i2ful3:
ZooKeeper ZOOKEEPER-2210

clock_gettime is not available in os x

Bug Resolved Major Fixed Michi Mutsuzaki Michi Mutsuzaki Michi Mutsuzaki 09/Jun/15 00:47   22/Jun/15 06:46 21/Jun/15 20:20   3.5.1, 3.6.0 c client   0 6   {noformat}
src/zookeeper.c:286:9: warning: implicit declaration of function 'clock_gettime' is invalid in C99 [-Wimplicit-function-declaration]
ret = clock_gettime(CLOCK_MONOTONIC, &ts);
^
src/zookeeper.c:286:23: error: use of undeclared identifier 'CLOCK_MONOTONIC'
ret = clock_gettime(CLOCK_MONOTONIC, &ts);
{noformat}
9223372036854775807 No Perforce job exists for this issue. 2 9223372036854775807
4 years, 39 weeks, 3 days ago 0|i2fsrj:
ZooKeeper ZOOKEEPER-2209

A .NET C# version of ZooKeeper client

New Feature Resolved Major Won't Fix Shay Hazor Shay Hazor Shay Hazor 08/Jun/15 03:19   13/Oct/16 16:38 31/Oct/15 03:24 3.4.6       0 6 0 0 0% .NET CoreCLR Inspired by the work of [~ewhauser] .
I propose a C# Client that supports the current stable version of ZK 3.4.6.
It was built by using static code conversion tools followed by manual editing and C# implementations of java selector and other java constructs.
A great measure was taken to follow the logic of the java version. In fact, the code is almost identical. Thus allowing easy evolution alongside the java version.

Main features:
* fully .NET async, no explicit threads used
* all relevant unit tests have been converted and passing consistently
* Code is 100% CoreCLR compliant
* [NuGet package|https://www.nuget.org/packages/ZooKeeperNetEx] is already integrated in [Microsoft Project Orleans|https://github.com/dotnet/orleans] as the only open-source membership provider.
* [Nuget package for recipes|https://www.nuget.org/packages/ZooKeeperNetEx.Recipes]
0% 0% 0 0 .NET, CoreCLR, async, c# 9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 23 weeks ago a ZooKeeper .NET async Client for ZK v3.4.6
Current Limitations:
* No support for system properties (currently the defaults are used).
* No SASL support
31 https://github.com/apache/zookeeper/pull/31 0|i2fqsn:
ZooKeeper ZOOKEEPER-2208

Log type of unexpected quorum packet in observer loop

Improvement Resolved Trivial Fixed Hitoshi Mitake Akihiro Suda Akihiro Suda 05/Jun/15 04:26   07/Jun/15 21:25 05/Jun/15 15:31 3.5.0 3.5.1, 3.6.0 server   0 3   This patch lets the observer loop log the type of packet for debugging.
This issue is tightly related to ZOOKEEPER-2205

9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
Patch
4 years, 41 weeks, 3 days ago 0|i2fo2f:
ZooKeeper ZOOKEEPER-2207

Enhance error logs with LearnerHandler.packetToString()

Improvement Resolved Trivial Fixed Hitoshi Mitake Hitoshi Mitake Hitoshi Mitake 04/Jun/15 03:52   07/Jun/15 02:04 05/Jun/15 15:18 3.5.0 3.5.1, 3.6.0 server   0 3   This patch enhances error logs related to unexpected types of QuorumPacket with LearnerHandler.packetToString(). 9223372036854775807 No Perforce job exists for this issue. 2 9223372036854775807
Patch
4 years, 41 weeks, 4 days ago 0|i2fmb3:
ZooKeeper ZOOKEEPER-2206

Add missing packet types to LearnerHandler.packetToString()

Improvement Resolved Trivial Fixed Hitoshi Mitake Hitoshi Mitake Hitoshi Mitake 04/Jun/15 03:44   07/Jun/15 02:04 05/Jun/15 15:01 3.5.0 3.5.1, 3.6.0 server   0 3   packetToString() is a method which is suitable for obtaining string representation of QuorumPacket. But it lacks some types of QuorumPacket. This patch adds the missing types and enhance the method for more friendly logging. 9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
Patch
4 years, 41 weeks, 4 days ago 0|i2fman:
ZooKeeper ZOOKEEPER-2205

Log type of unexpected quorum packet in learner handler loop

Improvement Resolved Trivial Fixed Hitoshi Mitake Hitoshi Mitake Hitoshi Mitake 04/Jun/15 01:44   07/Jun/15 02:04 05/Jun/15 14:45 3.4.6, 3.5.0 3.4.7, 3.5.1, 3.6.0 server   0 5   Current learner handler loop doesn't log anything when it receives unexpected type of quorum packet from learner.

This patch lets the learner handler loop log the type of packet for defensive purpose. It would make debugging and trouble shooting a little bit easier.
9223372036854775807 No Perforce job exists for this issue. 4 9223372036854775807
Patch
4 years, 41 weeks, 4 days ago 0|i2fm6n:
ZooKeeper ZOOKEEPER-2204

LearnerSnapshotThrottlerTest.testHighContentionWithTimeout fails occasionally

Test Resolved Minor Fixed Donny Nadolny Donny Nadolny Donny Nadolny 03/Jun/15 12:04   19/May/18 19:50 04/Jun/15 13:40 3.5.0 3.5.1, 3.6.0     0 5   The {{LearnerSnapshotThrottler}} will only allow 2 concurrent snapshots to be taken, and if there are already 2 snapshots in progress it will wait up to 200ms for one to complete. This isn't enough time for {{testHighContentionWithTimeout}} to consistently pass - on a cold JVM running just the one test I was able to get it to fail 3 times in around 50 runs. This 200ms timeout will be hit if there is a delay between a thread calling {{LearnerSnapshot snap = throttler.beginSnapshot(false);}} and {{throttler.endSnapshot();}}.
This also erroneously fails on the build server, see https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/2747/testReport/org.apache.zookeeper.server.quorum/LearnerSnapshotThrottlerTest/testHighContentionWithTimeout/ for an example.

I have bumped the timeout up to 5 seconds (which should be more than enough for warmup / gc pauses), as well as added logging to the {{catch (Exception e)}} block to assist in debugging any future issues.

An alternate approach would be to separate out results gathered from the threads, because although we only record true/false there are really three outcomes:
1. The {{snapshotNumber}} was <= 2, meaning the individual call operated correctly
2. The {{snapshotNumber}} was > 2, meaning the test should definitely fail
3. We were unable to snapshot in the time given, so we can't determine if we should fail or pass (although if we have "enough" successes from #1 with no failures from #2 maybe we would pass the test anyway).
Bumping up the timeout is easier.
9223372036854775807 No Perforce job exists for this issue. 2 9223372036854775807
Patch
4 years, 41 weeks, 6 days ago 0|i2fl1r:
ZooKeeper ZOOKEEPER-2203

multiple leaders can be elected when configs conflict

Bug Resolved Major Not A Problem Unassigned Akihiro Suda Akihiro Suda 02/Jun/15 23:36   16/Jun/15 14:17 16/Jun/15 14:17 3.5.0   leaderElection   0 2   This sequence leads the ensemble to a split-brain state:
* Start server 1 (config=1:participant, 2:participant, 3:participant)
* Start server 2 (config=1:participant, 2:participant, 3:participant)
* 1 and 2 believe 2 is the leader
* Start server 3 (config=1:observer, 2:observer, 3:participant)
* 3 believes 3 is the leader, although 1 and 2 still believe 2 is the leader

Such a split-brain ensemble is very unstable.
Znodes can be lost easily:
* Create some znodes on 2
* Restart 1 and 2
* 1, 2 and 3 can think 3 is the leader
* znodes created on 2 are lost, as 1 and 2 sync with 3


I consider this behavior as a bug and that ZK should fail gracefully if a participant is listed as an observer in the config.

In current implementation, ZK cannot detect such an invalid config, as FastLeaderElection.sendNotification() sends notifications to only voting members and hence there is no message from observers(1 and 2) to the new voter (3).
I think FastLeaderElection.sendNotification() should send notifications to all the members and FastLeaderElection.Messenger.WorkerReceiver.run() should verify acks.

Any thoughts?
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
4 years, 40 weeks, 2 days ago ZOOKEEPER-368 0|i2fk0f:
ZooKeeper ZOOKEEPER-2202

Cluster crashes when reconfig adds an unreachable observer

Bug Patch Available Major Unresolved Raúl Gutiérrez Segalés Raúl Gutiérrez Segalés Raúl Gutiérrez Segalés 02/Jun/15 17:44   05/Feb/20 07:11   3.5.0, 3.6.0 3.7.0, 3.5.8     0 7   While adding support for reconfig() in Kazoo (https://github.com/python-zk/kazoo/pull/333) I found that the cluster can be crashed if you add an observer whose election port isn't reachable (i.e.: packets for that destination are dropped, not rejected). This will raise a SocketTimeoutException which will bring down the PrepRequestProcessor:

{code}
2015-06-02 14:37:16,473 [myid:3] - WARN [ProcessThread(sid:3 cport:-1)::QuorumCnxManager@384] - Cannot open channel to 100 at election address /8.8.8.8:38703
java.net.SocketTimeoutException: connect timed out
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:345)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:369)
at org.apache.zookeeper.server.quorum.QuorumPeer.connectNewPeers(QuorumPeer.java:1288)
at org.apache.zookeeper.server.quorum.QuorumPeer.setLastSeenQuorumVerifier(QuorumPeer.java:1315)
at org.apache.zookeeper.server.quorum.Leader.propose(Leader.java:1056)
at org.apache.zookeeper.server.quorum.ProposalRequestProcessor.processRequest(ProposalRequestProcessor.java:78)
at org.apache.zookeeper.server.PrepRequestProcessor.pRequest(PrepRequestProcessor.java:877)
at org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:143)
{code}

A simple repro can be obtained by using the code in the referenced pull request above and using 8.8.8.8:3888 (for example) instead of a free (but closed) port in the loopback.

I think that adding an Observer (or a Participant) that isn't currently reachable is a valid use case (i.e.: you are provisioning the machine and it's not currently needed) so I think we could handle this with lower connect timeouts, not sure.
9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
1 year, 17 weeks, 1 day ago 0|i2fjgv:
ZooKeeper ZOOKEEPER-2201

Network issues can cause cluster to hang due to near-deadlock

Bug Closed Critical Fixed Donny Nadolny Donny Nadolny Donny Nadolny 01/Jun/15 21:30   21/Jul/16 16:18 06/Jun/15 12:54 3.4.6, 3.5.0 3.4.7, 3.5.2, 3.6.0     0 10   {{DataTree.serializeNode}} synchronizes on the {{DataNode}} it is about to serialize then writes it out via {{OutputArchive.writeRecord}}, potentially to a network connection. Under default linux TCP settings, a network connection where the other side completely disappears will hang (blocking on the {{java.net.SocketOutputStream.socketWrite0}} call) for over 15 minutes. During this time, any attempt to create/delete/modify the {{DataNode}} will cause the leader to hang at the beginning of the request processor chain:

{noformat}
"ProcessThread(sid:5 cport:-1):" prio=10 tid=0x00000000026f1800 nid=0x379c waiting for monitor entry [0x00007fe6c2a8c000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.apache.zookeeper.server.PrepRequestProcessor.getRecordForPath(PrepRequestProcessor.java:163)
- waiting to lock <0x00000000d4cd9e28> (a org.apache.zookeeper.server.DataNode)
- locked <0x00000000d2ef81d0> (a java.util.ArrayList)
at org.apache.zookeeper.server.PrepRequestProcessor.pRequest2Txn(PrepRequestProcessor.java:345)
at org.apache.zookeeper.server.PrepRequestProcessor.pRequest(PrepRequestProcessor.java:534)
at org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:131)
{noformat}

Additionally, any attempt to send a snapshot to a follower or to disk will hang.

Because the ping packets are sent by another thread which is unaffected, followers never time out and become leader, even though the cluster will make no progress until either the leader is killed or the TCP connection times out. This isn't exactly a deadlock since it will resolve itself eventually, but as mentioned above this will take > 15 minutes with the default TCP retry settings in linux.

A simple solution to this is: in {{DataTree.serializeNode}} we can take a copy of the contents of the {{DataNode}} (as is done with its children) in the synchronized block, then call {{writeRecord}} with the copy of the {{DataNode}} outside of the original {{DataNode}} synchronized block.
9223372036854775807 No Perforce job exists for this issue. 6 9223372036854775807
4 years, 41 weeks, 4 days ago 0|i2fhnj:
ZooKeeper ZOOKEEPER-2200

Perl ZooKeeper locks up during heavy load

Bug Open Major Unresolved Unassigned KARA VAN HORN KARA VAN HORN 01/Jun/15 15:02   30/Mar/16 12:42   3.4.6   c client   0 2   CentOS 5.8 We are using Perl Net::ZooKeeper (0.38) and Net::ZooKeeper::Lock (0.03) libraries. Deadlock appears to occur at the end during lock cleanup activity. Here is a stack dump (sensitive names changed):

Thread 2 (Thread 0x2ac6fbfa3940 (LWP 13292)):
#0 0x00002ac6f5aed654 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00002ac6f5aeb47b in pthread_cond_signal@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#2 0x00002ac6f835539c in _zk_watcher (handle=<value optimized out>, type=2, state=3, path=<value optimized out>, context=0x33f3ce0) at ZooKeeper.xs:179
#3 0x00002ac6f856d942 in do_foreach_watcher (zh=0x33e4fb0, type=2, state=3, path=0x33f3f50 "/lock/cmts/cisco_device1.net-0001851215", list=0x33ed290)
at /home/myhome/rpm/BUILD/zookeeper-3.4.6/src/c/src/zk_hashtable.c:279
#4 deliverWatchers (zh=0x33e4fb0, type=2, state=3, path=0x33f3f50 "/lock/cmts/cisco_device1.net-0001851215", list=0x33ed290)
at /home/myhome/rpm/BUILD/zookeeper-3.4.6/src/c/src/zk_hashtable.c:321
#5 0x00002ac6f8564966 in process_completions (zh=0x33e4fb0) at /home/myhome/rpm/BUILD/zookeeper-3.4.6/src/c/src/zookeeper.c:2114
#6 0x00002ac6f856e101 in do_completion (v=<value optimized out>) at /home/myhome/rpm/BUILD/zookeeper-3.4.6/src/c/src/mt_adaptor.c:466
#7 0x00002ac6f5ae683d in start_thread (arg=<value optimized out>) at pthread_create.c:301
#8 0x00002ac6f5dd1fcd in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x2ac6f6056af0 (LWP 12972)):
#0 0x00002ac6f5ae7c65 in pthread_join (threadid=47034119371072, thread_return=0x0) at pthread_join.c:89
#1 0x00002ac6f856e7de in adaptor_finish (zh=0x33e4fb0) at /home/myhome/rpm/BUILD/zookeeper-3.4.6/src/c/src/mt_adaptor.c:293
#2 0x00002ac6f8566cdc in zookeeper_close (zh=0x33e4fb0) at /home/myhome/rpm/BUILD/zookeeper-3.4.6/src/c/src/zookeeper.c:2536
#3 0x00002ac6f8357222 in XS_Net__ZooKeeper_DESTROY (my_perl=0x20df010, cv=<value optimized out>) at ZooKeeper.xs:885
#4 0x00002ac6f4b38af6 in Perl_pp_entersub () from /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so
#5 0x00002ac6f4adb8d7 in ?? () from /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so
#6 0x00002ac6f4adf720 in Perl_call_sv () from /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so
#7 0x00002ac6f4b3d3c6 in Perl_sv_clear () from /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so
#8 0x00002ac6f4b3db70 in Perl_sv_free () from /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so
#9 0x00002ac6f4b6025c in Perl_free_tmps () from /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so
#10 0x00002ac6f4adf78a in Perl_call_sv () from /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so
#11 0x00002ac6f4b3d3c6 in Perl_sv_clear () from /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so
#12 0x00002ac6f4b3db70 in Perl_sv_free () from /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so
#13 0x00002ac6f4b3b0e5 in ?? () from /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so
#14 0x00002ac6f4b3b141 in Perl_sv_clean_objs () from /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so
#15 0x00002ac6f4ae185e in perl_destruct () from /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so
#16 0x0000000000401773 in main ()

There are about 4 out of 10,000 processes that end up in deadlock, and according to our web searches, the only reason pthread_cond_signal would lock is due to it waiting on an already destroyed condition.
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 51 weeks, 1 day ago 0|i2fgzz:
ZooKeeper ZOOKEEPER-2199

Don't include unistd.h in windows

Bug Resolved Major Duplicate Michi Mutsuzaki Michi Mutsuzaki Michi Mutsuzaki 31/May/15 19:28   19/Dec/19 18:02 01/Jun/15 02:36   3.5.1 c client   0 2   Windows doesn't have unistd.h.

https://builds.apache.org/view/S-Z/view/ZooKeeper/job/ZooKeeper-trunk-WinVS2008/
9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
4 years, 42 weeks, 3 days ago 0|i2ffqf:
ZooKeeper ZOOKEEPER-2198

Set default test.junit.threads to 1.

Bug Resolved Minor Fixed Chris Nauroth Chris Nauroth Chris Nauroth 30/May/15 17:22   31/May/15 14:55 31/May/15 05:23   3.5.1, 3.6.0 build   0 4   Some systems are seeing test failures under concurrent execution. This issue proposes to change the default {{test.junit.threads}} to 1 so that those environments continue to get consistent test runs. Jenkins and individual developer environments can set multiple threads with a command line argument, so most environments will still get the benefit of faster test runs. 9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
4 years, 42 weeks, 4 days ago 0|i2ff3r:
ZooKeeper ZOOKEEPER-2197

non-ascii character in FinalRequestProcessor.java

Bug Resolved Minor Fixed Michi Mutsuzaki Michi Mutsuzaki Michi Mutsuzaki 30/May/15 16:25   02/Jun/15 06:57 02/Jun/15 01:25   3.5.1, 3.6.0     0 7   src/java/main/org/apache/zookeeper/server/FinalRequestProcessor.java:134: error: unmappable character for encoding ASCII
[javac] // was not being queued ??? ZOOKEEPER-558) properly. This happens, for example,
9223372036854775807 No Perforce job exists for this issue. 5 9223372036854775807
4 years, 42 weeks, 2 days ago 0|i2ff3b:
ZooKeeper ZOOKEEPER-2196

Web Access link broken

Bug Open Major Unresolved Unassigned Rob Eden Rob Eden 22/May/15 17:29   22/May/15 17:29           0 1   The link for "Web Access" on the [SVN site page|https://zookeeper.apache.org/svn.html] is broken.

This link:

http://svn.apache.org/viewcvs.cgi/zookeeper/

Should be replaced with this:

https://svn.apache.org/viewvc/zookeeper

(in "site/trunk/content/svn.textile")
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
4 years, 43 weeks, 6 days ago 0|i2f4jr:
ZooKeeper ZOOKEEPER-2195

fsync.warningthresholdms in zoo.cfg not working

Bug Closed Trivial Fixed Biju Nair David Fan David Fan 21/May/15 03:27   21/Jul/16 16:18 20/Mar/16 14:37 3.4.6, 3.5.0 3.4.9, 3.5.2, 3.6.0 quorum   0 5   Config fsync.warningthresholdms in zoo.cfg does not work.
I find QuorumPeerConfig.parseProperties give fsync.warningthresholdms a prefix like "zookeeper.fsync.warningthresholdms". But in class FileTxnLog where fsync.warningthresholdms is used, code is :Long.getLong("fsync.warningthresholdms", 1000),without prefix "zookeeper.", therefore can not get fsync.warningthresholdms's value.

I wonder the speed of fsync, need this config to see whether the speed is good enough.
9223372036854775807 No Perforce job exists for this issue. 4 9223372036854775807
4 years, 4 days ago
Reviewed
0|i2f16f:
ZooKeeper ZOOKEEPER-2194

Let DataNode.getChildren() return an unmodifiable view of its children set

Improvement Resolved Trivial Fixed Hitoshi Mitake Hitoshi Mitake Hitoshi Mitake 21/May/15 02:35   05/Jun/15 06:47 04/Jun/15 12:29 3.4.6, 3.5.0 3.4.7, 3.5.1, 3.6.0 server   0 5   Current DataNode.getChildren() directly returns a pointer of its private member, children. However, the member should be modified through addChild() and removeChild(). Callers of getChildren() shouldn't modify it directly.

For preventing the direct modification by the callers, this patch lets getChildren() return an unmodifiable view of its children set. If the callers try to modify directly, runtime exception will be risen.
9223372036854775807 No Perforce job exists for this issue. 2 9223372036854775807
Patch
4 years, 41 weeks, 6 days ago 0|i2f11r:
ZooKeeper ZOOKEEPER-2193

reconfig command completes even if parameter is wrong obviously

Bug Resolved Major Fixed Yasuhito Fukuda Yasuhito Fukuda Yasuhito Fukuda 20/May/15 05:38   28/Jul/15 02:49 26/Jun/15 20:06 3.5.0 3.5.1, 3.6.0 leaderElection, server   0 9   CentOS7 + Java7 Even if reconfig parameter is wrong, it was confirmed to complete.
refer to the following.

- Ensemble consists of four nodes

{noformat}
[zk: vm-101:2181(CONNECTED) 0] config
server.1=192.168.100.101:2888:3888:participant
server.2=192.168.100.102:2888:3888:participant
server.3=192.168.100.103:2888:3888:participant
server.4=192.168.100.104:2888:3888:participant
version=100000000
{noformat}

- add node by reconfig command

{noformat}
[zk: vm-101:2181(CONNECTED) 9] reconfig -add server.5=192.168.100.104:2888:3888:participant;0.0.0.0:2181
Committed new configuration:
server.1=192.168.100.101:2888:3888:participant
server.2=192.168.100.102:2888:3888:participant
server.3=192.168.100.103:2888:3888:participant
server.4=192.168.100.104:2888:3888:participant
server.5=192.168.100.104:2888:3888:participant;0.0.0.0:2181
version=300000007
{noformat}

server.4 and server.5 of the IP address is a duplicate.

In this state, reader election will not work properly.
Besides, it is assumed an ensemble will be undesirable state.
I think that need a parameter validation when reconfig.
9223372036854775807 No Perforce job exists for this issue. 8 9223372036854775807
4 years, 38 weeks, 3 days ago 0|i2ez9j:
ZooKeeper ZOOKEEPER-2192

ZOOKEEPER-2163 Port "Introduce new ZNode type: container" to 3.4.x

Sub-task Patch Available Major Unresolved Jordan Zimmerman Jordan Zimmerman Jordan Zimmerman 19/May/15 12:57   19/Mar/19 08:43   3.4.6   c client, java client, server   0 6   ZOOKEEPER-2163 applies to the trunk branch. This feature is too needed to wait for 3.5.x. So, port the feature to the 3.4.x branch so it can be released ahead of 3.5.x. container_znode_type 9223372036854775807 No Perforce job exists for this issue. 2 9223372036854775807
3 years, 33 weeks ago 0|i2exs7:
ZooKeeper ZOOKEEPER-2191

Continue supporting prior Ant versions that don't implement the threads attribute for the JUnit task.

Improvement Closed Major Fixed Chris Nauroth Chris Nauroth Chris Nauroth 14/May/15 14:32   21/Jul/16 16:18 22/May/15 01:28   3.5.2, 3.6.0 build   0 6   ZOOKEEPER-2183 introduced usage of the threads attribute on the <junit> task call in build.xml to speed up test execution. This attribute is only available since Ant 1.9.4. However, we can continue to support older Ant versions by calling the <antversion> task and dispatching to a clone of our <junit> task call that doesn't use the threads attribute. Users of older Ant versions will get the slower single-process test execution. 9223372036854775807 No Perforce job exists for this issue. 2 9223372036854775807
4 years, 43 weeks, 6 days ago 0|i2eqs7:
ZooKeeper ZOOKEEPER-2190

In StandaloneDisabledTest, testReconfig() shouldn't take leaving servers as joining servers

Bug Resolved Major Fixed Hongchao Deng Hongchao Deng Hongchao Deng 13/May/15 22:59   16/May/15 00:20 14/May/15 15:57   3.5.1, 3.6.0 tests   0 6   9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
4 years, 44 weeks, 6 days ago trunk: http://svn.apache.org/viewvc?view=revision&revision=1679444
branch-3.5: http://svn.apache.org/viewvc?view=revision&revision=1679446
0|i2epp3:
ZooKeeper ZOOKEEPER-2189

QuorumCnxManager: use BufferedOutputStream for initial msg

Bug Open Major Unresolved Unassigned Akihiro Suda Akihiro Suda 13/May/15 02:20   03/Jun/15 12:46   3.5.0   leaderElection   0 5   This was original JIRA of ZOOKEEPER-2203. For project management reason, all the issues and related discussion are moved to ZOOKEEPER-2203. This JIRA is linked to ZOOKEEPER-2098.

==============

This sequence leads the ensemble to a split-brain state:
* Start server 1 (config=1:participant, 2:participant, 3:participant)
* Start server 2 (config=1:participant, 2:participant, 3:participant)
* 1 and 2 believe 2 is the leader
* Start server 3 (config=1:observer, 2:observer, 3:participant)
* 3 believes 3 is the leader, although 1 and 2 still believe 2 is the leader

Such a split-brain ensemble is very unstable.
Znodes can be lost easily:
* Create some znodes on 2
* Restart 1 and 2
* 1, 2 and 3 can think 3 is the leader
* znodes created on 2 are lost, as 1 and 2 sync with 3


I consider this behavior as a bug and that ZK should fail gracefully if a participant is listed as an observer in the config.

In current implementation, ZK cannot detect such an invalid config, as FastLeaderElection.sendNotification() sends notifications to only voting members and hence there is no message from observers(1 and 2) to the new voter (3).
I think FastLeaderElection.sendNotification() should send notifications to all the members and FastLeaderElection.Messenger.WorkerReceiver.run() should verify acks.

Any thoughts?
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
4 years, 42 weeks, 1 day ago ZOOKEEPER-368 0|i2eny7:
ZooKeeper ZOOKEEPER-2188

client connection hung up because of dead loop

Bug Open Major Unresolved Unassigned sunhaitao sunhaitao 11/May/15 07:07   30/Jul/15 01:31   3.5.0   java client   0 3   There is something wrong with the client code ClientCnxn.java, it will keep trying to connect to server in a dead loop.
This is my test step, shut down zookeeper cluster, exectue zkCli.sh script to connect to zookeeper cluster, it will keep trying to connect to zookeeper server without stop.

public void run() {
clientCnxnSocket.introduce(this, sessionId, outgoingQueue);
clientCnxnSocket.updateNow();
clientCnxnSocket.updateLastSendAndHeard();
int to;
long lastPingRwServer = Time.currentElapsedTime();
final int MAX_SEND_PING_INTERVAL = 10000; //10 seconds
while (state.isAlive()) {
try {
if (!clientCnxnSocket.isConnected()) {
// don't re-establish connection if we are closing
if (closing) {
break;
}
startConnect();
clientCnxnSocket.updateLastSendAndHeard();
}

public boolean isAlive() {
return this != CLOSED && this != AUTH_FAILED;
}

because at the beginning it is CONNECTING so isAlive always returns true, which leads to dead loop.
we should add some retry limit to stop this
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
4 years, 34 weeks ago 0|i2ejtz:
ZooKeeper ZOOKEEPER-2187

remove duplicated code between CreateRequest{,2}

Bug Resolved Minor Fixed Raúl Gutiérrez Segalés Raúl Gutiérrez Segalés Raúl Gutiérrez Segalés 08/May/15 23:53   30/May/15 06:40 29/May/15 13:47   3.5.1, 3.6.0 c client, java client, server   1 6   To avoid cargo culting and reducing duplicated code we can merge most of CreateRequest & CreateRequest2 given that only the Response object is actually different.

This will improve readability of the code plus make it less confusing for people adding new opcodes in the future (i.e.: copying a request definition vs reusing what's already there, etc.).
9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
4 years, 42 weeks, 5 days ago 0|i2eigv:
ZooKeeper ZOOKEEPER-2186

QuorumCnxManager#receiveConnection may crash with random input

Bug Resolved Major Fixed Raúl Gutiérrez Segalés Raúl Gutiérrez Segalés Raúl Gutiérrez Segalés 08/May/15 15:36   09/Apr/18 06:11 24/May/15 02:34 3.4.6, 3.5.0 3.4.7, 3.5.1, 3.6.0 server   0 13   This will allocate an arbitrarily large byte buffer (and try to read it!):

{code}
public boolean receiveConnection(Socket sock) {
Long sid = null;
...
sid = din.readLong();
// next comes the #bytes in the remainder of the message
int num_remaining_bytes = din.readInt();
byte[] b = new byte[num_remaining_bytes];
// remove the remainder of the message from din
int num_read = din.read(b);
{code}

This will crash the QuorumCnxManager thread, so the cluster will keep going but future elections might fail to converge (ditto for leaving/joining members).

Patch coming up in a bit.
9223372036854775807 No Perforce job exists for this issue. 4 9223372036854775807
4 years, 10 weeks, 2 days ago 0|i2ehxz:
ZooKeeper ZOOKEEPER-2185

Run server with -XX:+HeapDumpOnOutOfMemoryError and -XX:OnOutOfMemoryError='kill %p'.

Improvement Resolved Minor Fixed Chris Nauroth Chris Nauroth Chris Nauroth 08/May/15 13:44   19/Jun/15 16:30 18/Jun/15 15:26   3.5.1, 3.6.0 documentation, scripts   0 3   Continuing to run a server process after it runs out of memory can lead to unexpected behavior. This issue proposes that we update scripts and documentation to add these JVM options:
# {{-XX:+HeapDumpOnOutOfMemoryError}} for help with post-mortem analysis of why the process ran out of memory.
# {{-XX:OnOutOfMemoryError='kill %p'}} to kill the JVM process, under the assumption that a process monitor will restart it.
9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
4 years, 39 weeks, 6 days ago 0|i2eht3:
ZooKeeper ZOOKEEPER-2184

Zookeeper Client should re-resolve hosts when connection attempts fail

Bug Closed Blocker Fixed Andor Molnar Robert P. Thille Robert P. Thille 07/May/15 19:46   30/Jan/19 09:47 22/Jun/18 18:11 3.4.6, 3.4.7, 3.4.8, 3.4.9, 3.4.10, 3.5.0, 3.5.1, 3.5.2, 3.5.3, 3.4.11 3.6.0, 3.4.13, 3.5.5 java client   16 40 0 39000   Ubuntu 14.04 host, Docker containers for Zookeeper & Kafka Testing in a Docker environment with a single Kafka instance using a single Zookeeper instance. Restarting the Zookeeper container will cause it to receive a new IP address. Kafka will never be able to reconnect to Zookeeper and will hang indefinitely. Updating DNS or /etc/hosts with the new IP address will not help the client to reconnect as the zookeeper/client/StaticHostProvider resolves the connection string hosts at creation time and never re-resolves.

A solution would be for the client to notice that connection attempts fail and attempt to re-resolve the hostnames in the connectString.
100% 100% 39000 0 easyfix, patch, pull-request-available 9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
1 year, 35 weeks, 3 days ago 0|i2eg9z:
ZooKeeper ZOOKEEPER-2183

Concurrent Testing Processes and Port Assignments

Improvement Resolved Major Fixed Chris Nauroth Chris Nauroth Chris Nauroth 07/May/15 19:12   18/May/15 13:54 14/May/15 12:45 3.5.0 3.5.1, 3.6.0 tests   0 7   Tests use {{PortAssignment#unique}} for assignment of the ports to bind during tests. Currently, this method works by using a monotonically increasing counter from a static starting point. Generally, this is sufficient to achieve uniqueness within a single JVM process, but it does not achieve uniqueness across multiple processes on the same host. This can cause tests to get bind errors if there are multiple pre-commit jobs running concurrently on the same Jenkins host. This also prevents running tests in parallel to improve the speed of pre-commit runs. 9223372036854775807 No Perforce job exists for this issue. 6 9223372036854775807
4 years, 44 weeks, 3 days ago 0|i2eg7r:
ZooKeeper ZOOKEEPER-2182

Several test suites are not running during pre-commit, because their names do not end with "Test".

Bug Resolved Major Fixed Chris Nauroth Chris Nauroth Chris Nauroth 07/May/15 14:23   12/May/15 01:47 12/May/15 01:37 3.5.0 3.5.1, 3.6.0 tests   0 6   In build.xml, the {{<junit>}} task definition uses an include pattern of {{\*\*/\*$\{test.category\}Test.java}}. This is important so that we don't accidentally try to run utility classes like {{PortAssignment}} or {{TestableZooKeeper}} as if they were JUnit suites. However, several test suites are misnamed so that they don't satisfy this pattern, and therefore pre-commit hasn't been running them.

{{ClientRetry}}
{{ReconfigFailureCases}}
{{WatchEventWhenAutoReset}}
9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
4 years, 45 weeks, 2 days ago 0|i2efkn:
ZooKeeper ZOOKEEPER-2181

Slider-Zookeeper integration testcase fails with Zookeeper-3.5.0-alpha version

Bug Open Major Unresolved Unassigned Ayappan Ayappan 05/May/15 07:16   03/Jun/15 07:07   3.5.0       0 1   TestZKIntegration testcase in slider fails with zookeeper-3.5.0-alpha version. From the logs, it came to know the state change went to LOST rather than CONNECTED while creating ZK path. The above testcase passes with zookeeper-3.4.6. A slider jira SLIDER-862 is already opened for this. But the problem seems to be with zookeeper-3.5.0-alpha.

Running org.apache.slider.common.tools.TestZKIntegration
2015-04-24 06:56:52,118 [Thread-2] INFO services.MicroZookeeperService (MicroZookeeperService.java:serviceStart(235)) - Starting Local Zookeeper service
2015-04-24 06:56:52,299 [Thread-2] INFO services.MicroZookeeperService (MicroZookeeperService.java:serviceStart(241)) - In memory ZK started at localhost:50577

2015-04-24 06:56:52,300 [Thread-2] INFO test.MicroZKCluster (MicroZKCluster.groovy:createCluster(53)) - Created Micro ZK cluster as localhost:50577
2015-04-24 06:56:52,492 [Thread-2] INFO imps.CuratorFrameworkImpl (CuratorFrameworkImpl.java:start(223)) - Starting
2015-04-24 06:56:52,513 [Thread-2] DEBUG zk.ZKIntegration (ZKIntegration.java:init(96)) - Binding ZK client to localhost:50577
2015-04-24 06:56:52,513 [Thread-2] INFO zk.BlockingZKWatcher (BlockingZKWatcher.java:waitForZKConnection(57)) - waiting for ZK event
2015-04-24 06:56:52,543 [Thread-2-EventThread] DEBUG zk.ZKIntegration (ZKIntegration.java:process(178)) - WatchedEvent state:Expired type:None path:null
2015-04-24 06:56:52,544 [Thread-2-EventThread] DEBUG zk.ZKIntegration (ZKIntegration.java:maybeInit(191)) - initing
2015-04-24 06:56:52,544 [Thread-2-EventThread] DEBUG zk.ZKIntegration (ZKIntegration.java:createPath(222)) - Creating ZK path /services
2015-04-24 06:56:52,545 [Thread-2-EventThread] INFO state.ConnectionStateManager (ConnectionStateManager.java:postState(194)) - State change: LOST
2015-04-24 06:56:52,546 [Thread-2-EventThread] WARN curator.ConnectionState (ConnectionState.java:handleExpiredSession(289)) - Session expired event received
2015-04-24 06:56:52,548 [ConnectionStateManager-0] WARN state.ConnectionStateManager (ConnectionStateManager.java:processEvents(212)) - There are no ConnectionStateListeners registered.
2015-04-24 06:56:52,549 [NIOWorkerThread-1] WARN server.NIOServerCnxn (NIOServerCnxn.java:doIO(368)) - Unable to read additional data from client sessionid 0x14ceb499c750000, likely client has closed socket
2015-04-24 06:56:52,550 [Thread-2-EventThread] ERROR zk.ZKIntegration (ZKIntegration.java:process(182)) - Failed to init
org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /services
at org.apache.zookeeper.KeeperException.create(KeeperException.java:131)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:1067)
at org.apache.slider.core.zk.ZKIntegration.createPath(ZKIntegration.java:223)
at org.apache.slider.core.zk.ZKIntegration.mkPath(ZKIntegration.java:242)
at org.apache.slider.core.zk.ZKIntegration.maybeInit(ZKIntegration.java:193)
at org.apache.slider.core.zk.ZKIntegration.process(ZKIntegration.java:180)
at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:539)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:515)
2015-04-24 06:56:52,550 [NIOWorkerThread-3] WARN server.NIOServerCnxn (NIOServerCnxn.java:doIO(368)) - Unable to read additional data from client sessionid 0x14ceb499c750001, likely client has closed socket
2015-04-24 06:56:52,551 [Thread-2-EventThread] INFO zk.BlockingZKWatcher (BlockingZKWatcher.java:process(37)) - ZK binding callback received
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
4 years, 46 weeks, 2 days ago 0|i2eaq7:
ZooKeeper ZOOKEEPER-2180

quota do not take effect in version 3.4.6

Bug Open Major Unresolved Unassigned seekerak seekerak 04/May/15 22:04   23/Mar/19 04:44           0 3   zookeeper version 3.4.6 [zk: localhost:2181(CONNECTED) 18] listquota /mynode
absolute path is /zookeeper/quota/mynode/zookeeper_limits
Output quota for /mynode count=-1,bytes=100
Output stat for /mynode count=6,bytes=484



[zk: localhost:2181(CONNECTED) 19] listquota /mynode_n
absolute path is /zookeeper/quota/mynode_n/zookeeper_limits
Output quota for /mynode_n count=2,bytes=-1
Output stat for /mynode_n count=5,bytes=5
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
51 weeks, 5 days ago 0|i2ea2n:
ZooKeeper ZOOKEEPER-2179

Typo in Watcher.java

Improvement Resolved Trivial Fixed Archana T Eunchan Kim Eunchan Kim 04/May/15 04:47   02/Mar/16 20:31 29/May/15 15:53 3.4.5, 3.5.0 3.4.7, 3.5.0, 3.6.0 server   0 5   at zookeeper/src/java/main/org/apache/zookeeper/Watcher.java,

* implement. A ZooKeeper client will get various events from the ZooKeepr

should be fixed to

* implement. A ZooKeeper client will get various events from the ZooKeeper.
(Zookeepr -> Zookeeper)

9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
4 years, 42 weeks, 5 days ago 0|i2e8k7:
ZooKeeper ZOOKEEPER-2178

Native client fails compilation on Windows.

Bug Resolved Major Fixed Chris Nauroth Chris Nauroth Chris Nauroth 02/May/15 16:29   14/Mar/17 00:32 01/Jun/15 02:48 3.5.0 3.5.1, 3.6.0 c client   0 6   Windows Due to several recent changes, the native client fails to compile on Windows:
# ZOOKEEPER-827 (read-only mode) mismatched a function return type between the declaration and definition.
# ZOOKEEPER-1626 (monotonic clock for tolerance to time adjustments) added an include of unistd.h, which does not exist on Windows.
# Additionally, ZOOKEEPER-1626 did not implement a code path for accessing the Windows monotonic clock.
9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
3 years, 1 week, 2 days ago 0|i2e7sn:
ZooKeeper ZOOKEEPER-2177

point to md5/sha1/asc files in releases.html

Task Open Minor Unresolved Chris Nauroth Michi Mutsuzaki Michi Mutsuzaki 30/Apr/15 02:26   18/Nov/15 20:05           0 3   these files are not mirrored. we should link to these files in http://zookeeper.apache.org/releases.html 9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
4 years, 18 weeks ago 0|i2e4in:
ZooKeeper ZOOKEEPER-2176

Unclear error message should be info not error

Improvement Resolved Major Fixed Raúl Gutiérrez Segalés Raúl Gutiérrez Segalés Raúl Gutiérrez Segalés 28/Apr/15 13:19   06/May/15 07:47 05/May/15 13:18 3.5.0 3.5.1, 3.6.0 quorum   0 4   Hi [~shralex],

Looking at the CI output of ZOOKEEPER-2163 I see this:

{noformat}
[exec] [junit] 2015-04-17 17:36:23,750 [myid:] - ERROR [QuorumPeer[myid=4](plain=/0:0:0:0:0:0:0:0:11235)(secure=disabled):QuorumPeer@1394] - writeToDisk == true but configFilename == null
{noformat}

Though looking at QuorumPeer#setQuorumVerifier I see:

{noformat}
if (configFilename != null) {
try {
String dynamicConfigFilename = makeDynamicConfigFilename(
qv.getVersion());
QuorumPeerConfig.writeDynamicConfig(
dynamicConfigFilename, qv, false);
QuorumPeerConfig.editStaticConfig(configFilename,
dynamicConfigFilename,
needEraseClientInfoFromStaticConfig());
} catch (IOException e) {
LOG.error("Error closing file: ", e.getMessage());
}
} else {
LOG.error("writeToDisk == true but configFilename == null");
}
{noformat}

there's no proper error handling so I guess maybe we should just make it a warning? Thoughts?
9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
4 years, 46 weeks, 1 day ago 0|i2e0uv:
ZooKeeper ZOOKEEPER-2175

Checksum validation for malformed packets needs to handle.

Bug Open Major Unresolved Unassigned Brahma Reddy Battula Brahma Reddy Battula 23/Apr/15 22:17   07/Apr/16 03:45           0 11   *Session Id from ZK :*

2015-04-15 21:24:54,257 | INFO | CommitProcessor:22 | Established session 0x164cb2b3e4b36ae4 with negotiated timeout 45000 for client /160.149.0.117:44586 | org.apache.zookeeper.server.ZooKeeperServer.finishSessionInit(ZooKeeperServer.java:623)
2015-04-15 21:24:54,261 | INFO | NIOServerCxn.Factory:160-149-0-114/160.149.0.114:24002 | Successfully authenticated client: authenticationID=hdfs/hadoop@HADOOP.COM; authorizationID=hdfs/hadoop@HADOOP.COM. | org.apache.zookeeper.server.auth.SaslServerCallbackHandler.handleAuthorizeCallback(SaslServerCallbackHandler.java:118)
2015-04-15 21:24:54,261 | INFO | NIOServerCxn.Factory:160-149-0-114/160.149.0.114:24002 | Setting authorizedID: hdfs/hadoop@HADOOP.COM | org.apache.zookeeper.server.auth.SaslServerCallbackHandler.handleAuthorizeCallback(SaslServerCallbackHandler.java:134)
2015-04-15 21:24:54,261 | INFO | NIOServerCxn.Factory:160-149-0-114/160.149.0.114:24002 | adding SASL authorization for authorizationID: hdfs/hadoop@HADOOP.COM | org.apache.zookeeper.server.ZooKeeperServer.processSasl(ZooKeeperServer.java:1009)
2015-04-15 21:24:54,262 | INFO | ProcessThread(sid:22 cport:-1): | Got user-level KeeperException when processing *{color:red}sessionid:0x164cb2b3e4b36ae4{color}* type:create cxid:0x3 zxid:0x20009fafc txntype:-1 reqpath:n/a Error Path:/hadoop-ha/hacluster/ActiveStandbyElectorLock Error:KeeperErrorCode = NodeExists for /hadoop-ha/hacluster/ActiveStandbyElectorLock | org.apache.zookeeper.server.PrepRequestProcessor.pRequest(PrepRequestProcessor.java:648)

*ZKFC Received :* ZK client

2015-04-15 21:24:54,237 | INFO | main-SendThread(160-149-0-114:24002) | Socket connection established to 160-149-0-114/160.149.0.114:24002, initiating session | org.apache.zookeeper.ClientCnxn$SendThread.primeConnection(ClientCnxn.java:854)
2015-04-15 21:24:54,257 | INFO | main-SendThread(160-149-0-114:24002) | Session establishment complete on server 160-149-0-114/160.149.0.114:24002, *{color:blue}sessionid = 0x144cb2b3e4b36ae4 {color}* , negotiated timeout = 45000 | org.apache.zookeeper.ClientCnxn$SendThread.onConnected(ClientCnxn.java:1259)
2015-04-15 21:24:54,260 | INFO | main-EventThread | EventThread shut down | org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:512)
2015-04-15 21:24:54,262 | INFO | main-EventThread | Session connected. | org.apache.hadoop.ha.ActiveStandbyElector.processWatchEvent(ActiveStandbyElector.java:547)
2015-04-15 21:24:54,264 | INFO | main-EventThread | Successfully authenticated to ZooKeeper using SASL. | org.apache.hadoop.ha.ActiveStandbyElector.processWatchEvent(ActiveStandbyElector.java:573)

one bit corrupted..please check the following for same..

144cb2b3e4b36ae4=1010001001100101100101011001111100100101100110110101011100100
164cb2b3e4b36ae4=1011001001100101100101011001111100100101100110110101011100100
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 50 weeks ago 0|i2dp53:
ZooKeeper ZOOKEEPER-2174

JUnit4ZKTestRunner logs test failure for all exceptions even if the test method is annotated with an expected exception.

Bug Closed Minor Fixed Chris Nauroth Chris Nauroth Chris Nauroth 21/Apr/15 19:43   21/Jul/16 16:18 03/May/15 14:04   3.4.7, 3.5.2, 3.6.0 tests   0 6   {{JUnit4ZKTestRunner}} wraps JUnit test method execution, and if any exception is thrown, it logs a message stating that the test failed. However, some ZooKeeper tests are annotated with {{@Test(expected=...)}} to indicate that an exception is the expected result, and thus the test passes. The runner should be aware of expected exceptions and only log if an unexpected exception occurs. 9223372036854775807 No Perforce job exists for this issue. 5 9223372036854775807
4 years, 46 weeks, 3 days ago 0|i2dkkn:
ZooKeeper ZOOKEEPER-2173

ZK startup failure should be handled with proper error message

Bug Resolved Major Fixed J.Andreina J.Andreina J.Andreina 21/Apr/15 06:34   28/Apr/15 00:30 27/Apr/15 20:39   3.5.1, 3.6.0     0 4   If any failure during zk Startup (myid file does not exist), then still zk startup returns as successful (STARTED).

ZK startup failure should be handled with proper error message
9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
4 years, 47 weeks, 2 days ago 0|i2dj8f:
ZooKeeper ZOOKEEPER-2172

Cluster crashes when reconfig a new node as a participant

Bug Closed Critical Fixed Mohammad Arshad Ziyou Wang Ziyou Wang 21/Apr/15 00:14   07/Mar/18 09:37 08/Sep/16 17:01 3.5.0 3.5.3, 3.6.0 leaderElection, quorum, server   0 16   Ubuntu 12.04 + java 7 The operations are quite simple: start three zk servers one by one, then reconfig the cluster to add the new one as a participant. When I add the third one, the zk cluster may enter a weird state and cannot recover.

I found “2015-04-20 12:53:48,236 [myid:1] - INFO [ProcessThread(sid:1 cport:-1)::PrepRequestProcessor@547] - Incremental reconfig” in node-1 log. So the first node received the reconfig cmd at 12:53:48. Latter, it logged “2015-04-20 12:53:52,230 [myid:1] - ERROR [LearnerHandler-/10.0.0.2:55890:LearnerHandler@580] - Unexpected exception causing shutdown while sock still open” and “2015-04-20 12:53:52,231 [myid:1] - WARN [LearnerHandler-/10.0.0.2:55890:LearnerHandler@595] - ******* GOODBYE /10.0.0.2:55890 ********”. From then on, the first node and second node rejected all client connections and the third node didn’t join the cluster as a participant. The whole cluster was done.

When the problem happened, all three nodes just used the same dynamic config file zoo.cfg.dynamic.10000005d which only contained the first two nodes. But there was another unused dynamic config file in node-1 directory zoo.cfg.dynamic.next which already contained three nodes.

When I extended the waiting time between starting the third node and reconfiguring the cluster, the problem didn’t show again. So it should be a race condition problem.
9223372036854775807 No Perforce job exists for this issue. 34 9223372036854775807
2 years, 2 weeks, 1 day ago
Reviewed
0|i2disv:
ZooKeeper ZOOKEEPER-2171

avoid reverse lookups in QuorumCnxManager

Bug Resolved Major Fixed Raúl Gutiérrez Segalés Raúl Gutiérrez Segalés Raúl Gutiérrez Segalés 20/Apr/15 20:22   15/Feb/16 06:59 09/May/15 18:31   3.5.1, 3.6.0 quorum   0 7   Apparently, ZOOKEEPER-107 (via a quick git-blame look) introduced a bunch of getHostName() calls in QCM. Besides the overhead, these can cause problems when mixed with failing/mis-configured DNS servers.

It would be nice to reduce them, if that doesn't affect operational correctness.
9223372036854775807 No Perforce job exists for this issue. 2 9223372036854775807
4 years, 45 weeks, 4 days ago 0|i2dihj:
ZooKeeper ZOOKEEPER-2170

Zookeeper is not logging as per the configuration in log4j.properties

Bug Patch Available Major Unresolved Mohammad Arshad Mohammad Arshad Mohammad Arshad 20/Apr/15 07:19   05/Feb/20 07:11     3.7.0, 3.5.8     1 7   In conf/log4j.properties default root logger is
{code}
zookeeper.root.logger=INFO, CONSOLE
{code}

Changing root logger to bellow value or any other value does not change logging effect
{code}
zookeeper.root.logger=DEBUG, ROLLINGFILE
{code}
9223372036854775807 No Perforce job exists for this issue. 5 9223372036854775807
1 year, 17 weeks, 1 day ago 0|i2dh0v:
ZooKeeper ZOOKEEPER-2169

Enable creation of nodes with TTLs

New Feature Resolved Major Fixed Jordan Zimmerman Camille Fournier Camille Fournier 16/Apr/15 16:21   07/Jan/20 07:52 09/Oct/16 14:36 3.6.0 3.5.5 c client, java client, jute, server   8 20   ZOOKEEPER-2608, ZOOKEEPER-2609 As a user, I would like to be able to create a node that is NOT tied to a session but that WILL expire automatically if action is not taken by some client within a time window.

I propose this to enable clients interacting with ZK via http or other "thin clients" to create ephemeral-like nodes.

Some ideas for the design, up for discussion:

The node should support all normal ZK node operations including ACLs, sequential key generation, etc, however, it should not support the ephemeral flag. The node will be created with a TTL that is updated via a refresh operation.

The ZK quorum will watch this node similarly to the way that it watches for session liveness; if the node is not refreshed within the TTL, it will expire.

QUESTIONS:

1) Should we let the refresh operation set the TTL to a different base value?
2) If so, should the setting of the TTL to a new base value cause a watch to fire?
3) Do we want to allow these nodes to have children or prevent this similar to ephemeral nodes?

100% 1800 0 ttl_nodes 9223372036854775807 No Perforce job exists for this issue. 9 9223372036854775807
3 years, 19 weeks ago 0|i2dd9j:
ZooKeeper ZOOKEEPER-2168

ZOOKEEPER-2163 Add C APIs for new createContainer Methods

Sub-task Closed Major Fixed Balazs Meszaros Jordan Zimmerman Jordan Zimmerman 15/Apr/15 10:50   20/May/19 13:50 18/Mar/19 20:12 3.5.0 3.6.0, 3.5.5 c client   0 4 0 6000   ZOOKEEPER-2163 adds new client methods to create containers. These need to be exposed in the C client as well. 100% 100% 6000 0 container_znode_type, pull-request-available 9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
1 year, 2 days ago 0|i2daef:
ZooKeeper ZOOKEEPER-2167

Restarting current leader node sometimes results in a permanent loss of quorum

Bug Open Major Unresolved Unassigned Mike Lundy Mike Lundy 14/Apr/15 20:01   15/Apr/15 23:50   3.4.6       0 2   I'm seeing an issue where a restart of the current leader node results in a long-term / permanent loss of quorum (I've only waited 30 minutes, but it doesn't look like it's making any progress). Restarting the same instance _again_ seems to resolve the problem.

To me, this looks a lot like the issue described in https://issues.apache.org/jira/browse/ZOOKEEPER-1026, but I'm filing this separately for the moment in case I am wrong.

Notes on the attached log:
1) If you search for XXX in the log, you'll see where I've annotated it to include where the process was told to terminate, when it is reported to have completed that, and then the same for the start
2) To save you the trouble of figuring it out, here's the zkid <=> ip mapping:
zid=1, ip=10.20.0.19
zid=2, ip=10.20.0.18
zid=3, ip=10.20.0.20
zid=4, ip=10.20.0.21
zid=5, ip=10.20.0.22
3) It's important to note that this is log is during the process of a rolling service restart to remove an instance; in this case, zid #2 / 10.20.0.18 is the one being removed, so if you see a conspicuous silence from that service, that's why.
4) I've been unable to reproduce this problem _except_ during cluster size changes, so I suspect that may be related; it's also important to note that this test is going from 5 -> 4 (which means, since we remove one and then do a rolling restart, we are actually temporarily dropping to 3). I know this is not a recommended thing (this is more of a stress test). We have seen this same problem on larger cluster sizes, it just seems easier to reproduce it on smaller sizes.
5) The log starts roughly at the point 10.20.0.21 / zid=4 wins the election during the final quorum; zid=4 is the one whose shutdown triggers the problem.
9223372036854775807 No Perforce job exists for this issue. 2 9223372036854775807
4 years, 49 weeks ago 0|i2d9c7:
ZooKeeper ZOOKEEPER-2166

backupOldConfig() doesn't check for null

Bug Open Major Unresolved Unassigned Jordan Zimmerman Jordan Zimmerman 14/Apr/15 17:00   14/Apr/15 17:07   3.5.0   server   0 2   QuorumPeerConfig.backupOldConfig() should check if configFileStr is null or not and do nothing if it is null. This is currently breaking Apache Curator's TestingCluster. 9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
4 years, 49 weeks, 2 days ago 0|i2d8zz:
ZooKeeper ZOOKEEPER-2165

OSGi requires package "server.quorom.flexible" be exported

Bug Open Minor Unresolved Unassigned Simon Kitching Simon Kitching 14/Apr/15 07:42   23/Apr/15 04:42       quorum   0 3   Class QuoromPeer has a constructor which takes a QuorumVerifier value as a parameter. This class is defined in package "org.apache.zookeeper.server.quorum.flexible" but that package is not exported. 9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
4 years, 48 weeks ago 0|i2d82f:
ZooKeeper ZOOKEEPER-2164

fast leader election keeps failing

Bug Resolved Major Fixed Mate Szalay-Beko Michi Mutsuzaki Michi Mutsuzaki 14/Apr/15 03:36   16/Mar/20 22:59 12/Mar/20 09:52 3.4.5 3.7.0, 3.6.1, 3.5.8 leaderElection   10 26 0 28200   I have a 3-node cluster with sids 1, 2 and 3. Originally 2 is the leader. When I shut down 2, 1 and 3 keep going back to leader election. Here is what seems to be happening.

- Both 1 and 3 elect 3 as the leader.
- 1 receives votes from 3 and itself, and starts trying to connect to 3 as a follower.
- 3 doesn't receive votes for 5 seconds because connectOne() to 2 doesn't timeout for 5 seconds: https://github.com/apache/zookeeper/blob/41c9fcb3ca09cd3d05e59fe47f08ecf0b85532c8/src/java/main/org/apache/zookeeper/server/quorum/QuorumCnxManager.java#L346
- By the time 3 receives votes, 1 has given up trying to connect to 3: https://github.com/apache/zookeeper/blob/41c9fcb3ca09cd3d05e59fe47f08ecf0b85532c8/src/java/main/org/apache/zookeeper/server/quorum/Learner.java#L247

I'm using 3.4.5, but it looks like this part of the code hasn't changed for a while, so I'm guessing later versions have the same issue.
100% 100% 28200 0 pull-request-available 9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
2 days ago 0|i2d7r3:
ZooKeeper ZOOKEEPER-2163

Introduce new ZNode type: container

New Feature Resolved Major Fixed Jordan Zimmerman Jordan Zimmerman Jordan Zimmerman 13/Apr/15 14:19   07/Jan/20 07:50 20/Jun/15 20:06 3.5.0 3.5.1 c client, java client, server   6 22 0 600   ZOOKEEPER-2168, ZOOKEEPER-2192 BACKGROUND
============
A recurring problem for ZooKeeper users is garbage collection of parent nodes. Many recipes (e.g. locks, leaders, etc.) call for the creation of a parent node under which participants create sequential nodes. When the participant is done, it deletes its node. In practice, the ZooKeeper tree begins to fill up with orphaned parent nodes that are no longer needed. The ZooKeeper APIs don’t provide a way to clean these. Over time, ZooKeeper can become unstable due to the number of these nodes.

CURRENT SOLUTIONS
===================
Apache Curator has a workaround solution for this by providing the Reaper class which runs in the background looking for orphaned parent nodes and deleting them. This isn’t ideal and it would be better if ZooKeeper supported this directly.

PROPOSAL
=========
ZOOKEEPER-723 and ZOOKEEPER-834 have been proposed to allow EPHEMERAL nodes to contain child nodes. This is not optimum as EPHEMERALs are tied to a session and the general use case of parent nodes is for PERSISTENT nodes. This proposal adds a new node type, CONTAINER. A CONTAINER node is the same as a PERSISTENT node with the additional property that when its last child is deleted, it is deleted (and CONTAINER nodes recursively up the tree are deleted if empty).

CANONICAL USAGE
================

{code}
while ( true) { // or some reasonable limit
try {
zk.create(path, ...);
break;
} catch ( KeeperException.NoNodeException e ) {
try {
zk.createContainer(containerPath, ...);
} catch ( KeeperException.NodeExistsException ignore) {
}
}
}
{code}
100% 100% 6600 0 container_znode_type, pull-request-available 9223372036854775807 No Perforce job exists for this issue. 12 9223372036854775807
4 years, 39 weeks, 5 days ago 0|i2d6vb:
ZooKeeper ZOOKEEPER-2162

infinite exception loop occurs when dataDir is lost

Bug Patch Available Major Unresolved Akihiro Suda Akihiro Suda Akihiro Suda 13/Apr/15 00:33   20/Sep/17 04:03   3.5.0   server   1 7   This sequence leads server.1 and server.2 to infinite exception loop.

* Start server.1 and server.2 with the initial ensemble server.1=participant, server.2=observer.
In this time, acceptedEpoch\[i\] == currentEpoch\[i\] == 1 for i = 1, 2.
* Invoke reconfig so that acceptedEpoch\[i\] and currentEpoch\[i\] grows up to 2.
* Kill server.2
* Remove dataDir of server.2 excluding the myid file.
(In real production environments, both of confDir and dataDir can be lost due to reprovisioning)
* Start server.2
* server.1 and server.2 enters infinite exception loop.
The log (threshold is set to INFO in log4j.properties) size can reach > 100MB in 30 seconds.

AFAIK, the bug can be reproduced with ZooKeeper@f5fb50ed2591ba9a24685a227bb5374759516828 (Apr 7, 2015).

I made a Docker container so that people who are interested can reproduce the bug easily. (Sorry for no JUnit test right now)
{noformat}
$ docker run -i -t --rm akihirosuda/zookeeper-bug01
Reproducing the bug: infinite exception loop occurs when dataDir is lost
* Resetting
* Starting [1,2] with the initial ensemble [1]
* Sleeping for 3 seconds
* Invoking Reconfig [1]->[2]
* Sleeping for 3 seconds
* Killing server.2 (pid=10542)
* Sleeping for 3 seconds
* Resetting /zk02_data
* Starting server.2
* Sleeping for 30 seconds
/zk01_log: 81665114 bytes
The log dir is extremely large. Perhaps the bug was REPRODUCED!
/zk02_log: 23949367 bytes
The log dir is extremely large. Perhaps the bug was REPRODUCED!
* Exiting
{noformat}

h2. Log
h3. server.1
{noformat}
2015-04-13 03:48:17,624 [myid:1] - INFO [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):QuorumPeer@1022] - FOLLOWING
2015-04-13 03:48:17,624 [myid:1] - INFO [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):ZooKeeperServer@825] - minSessionTimeout set to 4000
2015-04-13 03:48:17,624 [myid:1] - INFO [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):ZooKeeperServer@834] - maxSessionTimeout set to 40000
2015-04-13 03:48:17,624 [myid:1] - INFO [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):ZooKeeperServer@156] - Created server with tickTime 2000 minSession
Timeout 4000 maxSessionTimeout 40000 datadir /zk01_data/version-2 snapdir /zk01_data/version-2
2015-04-13 03:48:17,624 [myid:1] - INFO [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):Follower@66] - FOLLOWING - LEADER ELECTION TOOK - 0
2015-04-13 03:48:17,625 [myid:1] - WARN [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):Follower@93] - Exception when following the leader
java.io.IOException: Leaders epoch, 1 is less than accepted epoch, 2
at org.apache.zookeeper.server.quorum.Learner.registerWithLeader(Learner.java:331)
at org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:75)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1024)
2015-04-13 03:48:17,626 [myid:1] - INFO [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):MBeanRegistry@119] - Unregister MBean [org.apache.ZooKeeperService:
name0=ReplicatedServer_id1,name1=replica.1,name2=Follower]
2015-04-13 03:48:17,626 [myid:1] - INFO [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):Follower@198] - shutdown called
java.lang.Exception: shutdown Follower
at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:198)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1028)
2015-04-13 03:48:17,626 [myid:1] - DEBUG [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):LearnerZooKeeperServer@162] - ZooKeeper server is not running, so n
ot proceeding to shutdown!
2015-04-13 03:48:17,626 [myid:1] - WARN [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):QuorumPeer@1071] - PeerState set to LOOKING
2015-04-13 03:48:17,626 [myid:1] - INFO [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):QuorumPeer@946] - LOOKING
2015-04-13 03:48:17,626 [myid:1] - DEBUG [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):QuorumPeer@875] - Initializing leader election protocol...
2015-04-13 03:48:17,626 [myid:1] - DEBUG [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):FastLeaderElection@790] - Updating proposal: -9223372036854775808 (
newleader), 0x100000002 (newzxid), -9223372036854775808 (oldleader), 0x100000002 (oldzxid)
2015-04-13 03:48:17,626 [myid:1] - INFO [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):FastLeaderElection@889] - New election. My id = 1, proposed zxid=0
x100000002
2015-04-13 03:48:17,626 [myid:1] - DEBUG [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):FastLeaderElection@673] - Sending Notification: -922337203685477580
8 (n.leader), 0x100000002 (n.zxid), 0x2 (n.round), 2 (recipient), 1 (myid), 0x2 (n.peerEpoch)
2015-04-13 03:48:17,626 [myid:1] - DEBUG [WorkerSender[myid=1]:QuorumCnxManager@400] - There is a connection already for server 2
2015-04-13 03:48:17,627 [myid:1] - DEBUG [WorkerReceiver[myid=1]:FastLeaderElection$Messenger$WorkerReceiver@336] - Receive new notification message. My id = 1
2015-04-13 03:48:17,627 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@683] - Notification: 2 (message format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round)
, LEADING (n.state), 2 (n.sid), 0x1 (n.peerEPoch), LOOKING (my state)100000002 (n.config version)
2015-04-13 03:48:17,627 [myid:1] - DEBUG [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):FastLeaderElection@812] - I'm a participant: 1
2015-04-13 03:48:17,627 [myid:1] - DEBUG [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):FastLeaderElection@637] - About to leave FLE instance: leader=2, zx
id=0x0, my id=1, my state=FOLLOWING
2015-04-13 03:48:17,627 [myid:1] - INFO [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):MBeanRegistry@119] - Unregister MBean [org.apache.ZooKeeperService:
name0=ReplicatedServer_id1,name1=replica.1,name2=LeaderElection]
2015-04-13 03:48:17,627 [myid:1] - INFO [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):QuorumPeer@1022] - FOLLOWING
..
{noformat}

h3. server.2
{noformat}
2015-04-13 03:48:17,672 [myid:2] - ERROR [LearnerHandler-/127.0.0.1:36337:LearnerHandler@580] - Unexpected exception causing shutdown while sock still open
java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:392)
at org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63)
at org.apache.zookeeper.server.quorum.QuorumPacket.deserialize(QuorumPacket.java:83)
at org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:99)
at org.apache.zookeeper.server.quorum.LearnerHandler.run(LearnerHandler.java:392)
2015-04-13 03:48:17,672 [myid:2] - WARN [LearnerHandler-/127.0.0.1:36337:LearnerHandler@595] - ******* GOODBYE /127.0.0.1:36337 ********
2015-04-13 03:48:17,674 [myid:2] - DEBUG [WorkerSender[myid=2]:QuorumCnxManager@400] - There is a connection already for server 1
2015-04-13 03:48:17,676 [myid:2] - INFO [LearnerHandler-/127.0.0.1:36338:LearnerHandler@364] - Follower sid: 1 not in the current config 100000002
2015-04-13 03:48:17,676 [myid:2] - ERROR [LearnerHandler-/127.0.0.1:36338:LearnerHandler@580] - Unexpected exception causing shutdown while sock still open
java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:392)
at org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63)
at org.apache.zookeeper.server.quorum.QuorumPacket.deserialize(QuorumPacket.java:83)
at org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:99)
at org.apache.zookeeper.server.quorum.LearnerHandler.run(LearnerHandler.java:392)
2015-04-13 03:48:17,677 [myid:2] - WARN [LearnerHandler-/127.0.0.1:36338:LearnerHandler@595] - ******* GOODBYE /127.0.0.1:36338 ********
..
{noformat}
9223372036854775807 No Perforce job exists for this issue. 5 9223372036854775807
2 years, 26 weeks, 1 day ago 0|i2d61b:
ZooKeeper ZOOKEEPER-2161

Cleanup task fails - java.io.FileNotFoundException: /zookeeper.log (Permission Denied)

Bug Open Major Unresolved Unassigned Michael Chiocca Michael Chiocca 08/Apr/15 12:10   08/Apr/15 12:10   3.4.6       0 1   The cleanup task fails with the following stack trace. This is happening repeatedly every time the cleanup task runs. Even the command line invocation of cleanup fails with the same stack trace.

zookeeper@zoo91-node-5dw4yocu7bvj-fpjhrmhvgyhz-mnjsb4zltcy5-7588:~$ java -cp ./zookeeper-3.4.6.jar:./lib/log4j-1.2.16.jar:./lib/slf4j-log4j12-1.6.1.jar:./lib/slf4j-api-1.6.1.jar:/etc/zookeeper/conf org.apache.zookeeper.server.PurgeTxnLog /var/log/zookeeper /var/lib/zookeeper 5
log4j:ERROR setFile(null,true) call failed.
java.io.FileNotFoundException: /zookeeper.log (Permission denied)
at java.io.FileOutputStream.open(Native Method)
at java.io.FileOutputStream.<init>(FileOutputStream.java:221)
at java.io.FileOutputStream.<init>(FileOutputStream.java:142)
at org.apache.log4j.FileAppender.setFile(FileAppender.java:294)
at org.apache.log4j.RollingFileAppender.setFile(RollingFileAppender.java:207)
at org.apache.log4j.FileAppender.activateOptions(FileAppender.java:165)
at org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:307)
at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:172)
at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:104)
at org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:809)
at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:735)
at org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:615)
at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:502)
at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:547)
at org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:483)
at org.apache.log4j.LogManager.<clinit>(LogManager.java:127)
at org.slf4j.impl.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:73)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:242)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:254)
at org.apache.zookeeper.server.PurgeTxnLog.<clinit>(PurgeTxnLog.java:45)

The data log dir is set to /var/log/zookeeper in the /etc/zookeeper/conf/zoo.cfg config file. But as you can see, specifying the config directory in the Java classpath doesn't help eliminate the problem.
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
4 years, 50 weeks, 1 day ago 0|i2czcn:
ZooKeeper ZOOKEEPER-2160

just test

Wish Resolved Trivial Invalid Unassigned zhu zhu 07/Apr/15 07:50   07/Apr/15 07:50 07/Apr/15 07:50         0 0   9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
4 years, 50 weeks, 2 days ago 0|i2cwkn:
ZooKeeper ZOOKEEPER-2159

Pluggable SASL Authentication

Improvement Open Major Unresolved Yuliya Feldman Yuliya Feldman Yuliya Feldman 07/Apr/15 02:59   05/Feb/20 07:16     3.7.0, 3.5.8 java client, server   9 13   Today SASLAuthenticationProvider is used for all SASL based authentications which creates some "if/else" statements in ZookeeperSaslClient and ZookeeperSaslServer code with just Kerberos and Digest.

We want to use yet another different SASL based authentication and adding one more "if/else" with some code specific just to that new way does not make much sense.

Proposal is to allow to plug custom SASL Authentication mechanism(s) without further changes in Zookeeper code.
9223372036854775807 No Perforce job exists for this issue. 2 9223372036854775807
3 years, 17 weeks, 1 day ago 0|i2cw2v:
ZooKeeper ZOOKEEPER-2158

CLONE - Switch to using maven to build ZooKeeper

Improvement Resolved Major Won't Do Hiram R. Chirino laesunK laesunK 06/Apr/15 10:31   04/Jul/18 11:07 04/Jul/18 11:07     build   0 2   Maven is a great too for building java projects at the ASF. It helps standardize the build a bit since it's a convention oriented.
It's dependency auto downloading would remove the need to store the dependencies in svn, and it will handle many of the suggested ASF policies like gpg signing of the releases and such.

The ZooKeeper build is almost vanilla except for the jute compiler bits. Things that would need to change are:
* re-organize the source tree a little so that it uses the maven directory conventions
* seperate the jute bits out into seperate modules so that a maven plugin can be with it

9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
1 year, 37 weeks, 1 day ago 0|i2cusn:
ZooKeeper ZOOKEEPER-2157

Upgrade option should be removed from zkServer.sh usage

Bug Resolved Minor Fixed J.Andreina J.Andreina J.Andreina 03/Apr/15 02:21   08/Apr/15 07:19 07/Apr/15 13:31   3.5.1, 3.6.0     0 5   Upgrade option should be removed from zkServer.sh usage from trunk code

Currently upgrade option is available in zkServer.sh usage , while upgrade feature is already been removed from trunk.

{noformat}
#:~/March_1/zookeeper/bin> ./zkServer.sh upgrade
ZooKeeper JMX enabled by default
Using config: /home/REX/March_1/zookeeper/bin/../conf/zoo.cfg
Usage: ./zkServer.sh [--config <conf-dir>] {start|start-foreground|stop|restart|status|upgrade|print-cmd}
{noformat}
9223372036854775807 No Perforce job exists for this issue. 2 9223372036854775807
4 years, 50 weeks, 1 day ago 0|i27qrz:
ZooKeeper ZOOKEEPER-2156

If JAVA_HOME is not set zk startup and fetching status command execution result misleads user.

Bug Closed Major Fixed J.Andreina J.Andreina J.Andreina 03/Apr/15 01:03   21/Jul/16 16:18 20/May/15 03:36   3.5.2, 3.6.0 scripts   0 6   If JAVA_HOME is not set, zk startup and fetching status command execution result misleads user.

1. Eventhough zk startup has failed since JAVA_HOME is not set , on CLI it displays that zk STARTED.
{noformat}
#:~/Apr3rd/zookeeper-3.4.6/bin> ./zkServer.sh start
JMX enabled by default
Using config: /home/REX/Apr3rd/zookeeper-3.4.6/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
{noformat}

2. Fetching zk status when JAVA_HOME is not set displays that process not running .
{noformat}
#:~/Apr3rd/zookeeper-3.4.6/bin> ./zkServer.sh status
JMX enabled by default
Using config: /home/REX/Apr3rd/zookeeper-3.4.6/bin/../conf/zoo.cfg
Error contacting service. It is probably not running.
{noformat}
9223372036854775807 No Perforce job exists for this issue. 5 9223372036854775807
4 years, 44 weeks, 1 day ago 0|i27qo7:
ZooKeeper ZOOKEEPER-2155

network is not good, the watcher in observer env will clear

Bug Resolved Critical Invalid Unassigned linking12 linking12 02/Apr/15 02:35   26/Aug/15 16:45 26/Aug/15 16:45 3.4.6 3.5.0 quorum   0 4   When I set up a ZooKeeper ensemble that uses Observers, The network is not very good.
I find all of the watcher disappear.

I read the source code and find:
When the observer connect to leader, will dump the DataTree from leader and rebuild in observer.
But the datawachers and childWatches is cleared for this operation.

after i change code like:
WatchManager dataWatchers = zk.getZKDatabase().getDataTree()
.getDataWatches();
WatchManager childWatchers = zk.getZKDatabase().getDataTree()
.getChildWatches();
zk.getZKDatabase().clear();
zk.getZKDatabase().deserializeSnapshot(leaderIs);
zk.getZKDatabase().getDataTree().setDataWatches(dataWatchers);
zk.getZKDatabase().getDataTree().setChildWatches(childWatchers);

The watcher do not disappear
moreinfo 9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
4 years, 30 weeks, 1 day ago 3.4.6 0|i27opr:
ZooKeeper ZOOKEEPER-2154

NPE in KeeperException

Bug Open Major Unresolved Unassigned Surendra Singh Lilhore Surendra Singh Lilhore 31/Mar/15 01:55   05/Feb/20 07:15   3.4.6 3.7.0, 3.5.8 java client   0 4   KeeperException should handle exception is code is null... 9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
1 year, 17 weeks, 1 day ago 0|i27kqn:
ZooKeeper ZOOKEEPER-2153

ZOOKEEPER-2120 X509 Authentication Documentation

Sub-task Resolved Major Fixed Ian Dimayuga Hongchao Deng Hongchao Deng 30/Mar/15 17:00   11/Sep/15 12:53 05/May/15 13:30 3.5.0 3.5.1, 3.6.0     0 5   9223372036854775807 No Perforce job exists for this issue. 3 9223372036854775807
4 years, 46 weeks ago 0|i27k5b:
ZooKeeper ZOOKEEPER-2152

ZOOKEEPER-2135 Intermittent failure in TestReconfig.cc

Sub-task Closed Major Fixed Michael Han Michi Mutsuzaki Michi Mutsuzaki 25/Mar/15 13:25   17/May/17 23:43 24/Aug/16 14:34   3.5.3, 3.6.0 c client   0 9   I'm seeing this failure in the c client test once in a while:

{noformat}
[exec] /home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk/trunk/src/c/tests/TestReconfig.cc:474: Assertion: assertion failed [Expression: found != string::npos, 10.10.10.4:2004 not in newComing list]
{noformat}

https://builds.apache.org/job/ZooKeeper-trunk/2640/console

reconfiguration 9223372036854775807 No Perforce job exists for this issue. 4 9223372036854775807
3 years, 30 weeks, 1 day ago
Reviewed
0|i27cmn:
ZooKeeper ZOOKEEPER-2151

FollowerZookeeperServer has thousands of outstanding requests stuck in CommitProcessor

Bug Resolved Major Duplicate Unassigned Jared Cantwell Jared Cantwell 24/Mar/15 15:20   25/Mar/15 16:35 25/Mar/15 16:35 3.5.0 3.5.0 server   0 2   Ubuntu 12.04 We are seeing one follower server in our quorum stuck with thousands of outstanding requests:
---------------------------------------------
node04:~$ telnet 10.10.10.6 2181
Trying 10.10.10.6...
Connected to 10.10.10.6.
Escape character is '^]'.
*stat*
Zookeeper version: 3.5.0-1547702, built on 05/15/2014 03:06 GMT
Clients:
/10.10.10.6:60646\[0\](queued=0,recved=1,sent=0)
/10.10.10.6:60648\[0\](queued=0,recved=1,sent=0)
/10.10.10.6:41786\[0\](queued=1,recved=3,sent=1)

Latency min/avg/max: 0/0/1887
Received: 3064156900
Sent: 3064134581
Connections: 3
*Outstanding: 24395*
Zxid: 0x11050f7e4b
Mode: follower
Node count: 6969
Connection closed by foreign host.
---------------------------------------------

When this happens, our c client is able to establish an initial connection to the server, but any request then times out. It re-establishes a connection, then times out, rinse, repeat. We are noticing this because we set up this particular client to connect directly to only one server in the quorum, so any problem with that server will be noticed. Our other clients are just connecting to the next server in the list, which is why only this client notices a problem.

We were able to capture a heap dump in one instance. This is what we observed:

- FollowerZookeeperServer.requestsInProcess has count ~24K
- CommitProcessor.queuedRequest list has the 24K items in it, so the FinalRequestProcessor's processRequest function isn't ever getting called to complete the requests.
- CommitProcessor.isWaitingForCommit()==true
- CommitProcessor.committedRequests.isEmpty()==true
- CommitProcessor.nextPending is a create request
- CommitProcessor.currentlyCommitting is null
- CommitProcessor.numRequestsProcessing is 0
- FollowerZookeeperServer, who should be calling commit() on the CommitProcessor, has no elements in its pendingTxns list, which indicates that it thinks it has already passed a COMMIT message to the CommitProcessor for every request that is stuck in the queuedRequests list and nextPending member of CommitProcessor.

The CommitProcessor's run() is doing this:
{quote}
Thread 23510: (state = BLOCKED)
java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be imprecise)
org.apache.zookeeper.server.quorum.CommitProcessor.run() @bci=165, line=182 (Compiled frame)
{quote}

When we attached via gdb to get the dump, sockets closed that caused a new round of leader election. When this happened, the issued corrected itself since the whole FollowerZookeeperServer got restarted.

I've confirmed that no time changing was happening before things got stuck 2 days before we noticed it.

9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
5 years, 1 day ago 0|i27aof:
ZooKeeper ZOOKEEPER-2150

Observers losing connection with active ensemble and dont recover

Bug Open Major Unresolved Unassigned parin jogani parin jogani 24/Mar/15 12:06   24/Mar/15 12:06   3.4.3   quorum   0 1   We have pool of zookeeper machines (contains both active and observer) in version 3.4.3.
We recently undated our exhibitor from 1.2.x to 1.5.4.
We are seeing a strange behavior in our observers: they keep losing connection with the active ensemble and do not recover. The connection goes into CLOSE_WAIT state. Dont think there is any relation to exhibitor.
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
5 years, 2 days ago 0|i27aan:
ZooKeeper ZOOKEEPER-2149

Logging of client address when socket connection established

Improvement Resolved Major Fixed Hongchao Deng Hongchao Deng Hongchao Deng 23/Mar/15 21:12   25/Mar/15 07:20 25/Mar/15 03:50   3.5.1, 3.6.0     0 3   When a socket connection is established, in ZooKeeperServer, it would print logs:

"Established session 0x$(session) with negotiated timeout $(timeout) for client: $(client_hostport)"

However, in client, it would only print the server address:
"Socket connection established to $(server_hostport), initiating session"

It would be nice to log client local address when socket connection established. Because clients will reconnect and ports is randomly assigned. We can better associate these addresses in this way.
9223372036854775807 No Perforce job exists for this issue. 2 9223372036854775807
5 years, 1 day ago 0|i2799z:
ZooKeeper ZOOKEEPER-2148

ZOOKEEPER-2120 ZooKeeper SSL User Guide

Sub-task Open Major Unresolved Hongchao Deng Hongchao Deng Hongchao Deng 22/Mar/15 19:33   02/Feb/16 00:59   3.5.1   java client   1 9   SSL is a new feature added in "3.5+".

We have a dedicated user guide:
https://cwiki.apache.org/confluence/display/ZOOKEEPER/ZooKeeper+SSL+User+Guide
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
4 years, 7 weeks, 2 days ago 0|i277hb:
ZooKeeper ZOOKEEPER-2147

Remove ZOO_NOTWATCHING_EVENT

Improvement Open Trivial Unresolved Unassigned Tom Distler Tom Distler 21/Mar/15 14:12   13/Apr/15 13:51   3.5.0   c client   0 1   All platforms A review of the ZK code shows that the NOTWATCHING event is never raised. However, most client users wouldn't know this and would (hopefully) write code to handle the event. We ran into this in our own code, as I refactored some event handling only to find this event isn't going to occur.

The responses from the community in the following discussion seem to confirm that this event should be removed: http://grokbase.com/t/zookeeper/user/1123dc333d/not-watching-event

I'm prepared to submit a patch (tested in our load-test environment) if this issue is accepted. I've removed the event completely from all client code (C, Python, etc). One possibility is to leave the event definition in-place, but add a "deprecated" comment so-as not to break existing code.
9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
4 years, 49 weeks, 3 days ago 0|i2736f:
ZooKeeper ZOOKEEPER-2146

BinaryInputArchive readString should check length before allocating memory

Bug Resolved Major Fixed Hongchao Deng Hongchao Deng Hongchao Deng 20/Mar/15 19:07   17/Apr/15 20:48 25/Mar/15 03:40   3.4.7, 3.5.1, 3.6.0     0 6   I recently observed a problem caused by malformed packets. ZK server crashed because of OutOfMemoryError.

The reason is BinaryInputArchive didn't check the length before allocating memory in readString():
{code}
public String readString(String tag) throws IOException {
int len = in.readInt();
if (len == -1) return null;
byte b[] = new byte[len];
...
{code}

I suggest to add the same check as in readBuffer.
9223372036854775807 No Perforce job exists for this issue. 5 9223372036854775807
4 years, 48 weeks, 5 days ago 0|i272m7:
ZooKeeper ZOOKEEPER-2145

Node can be seen but not deleted

Bug Open Major Unresolved Unassigned Frans Lawaetz Frans Lawaetz 18/Mar/15 16:27   18/Jan/16 19:40   3.4.6       0 6   I have a three-server ensemble that appears to be working fine in every respect but for the fact that I can ls or get a znode but can not rmr it.

>[zk: localhost:2181(CONNECTED) 0] get /accumulo/9354e975-7e2a-4207-8c7b-5d36c0e7765d/masters/goal_state
CLEAN_STOP
cZxid = 0x15
ctime = Fri Feb 20 13:37:59 CST 2015
mZxid = 0x72
mtime = Fri Feb 20 13:38:05 CST 2015
pZxid = 0x15
cversion = 0
dataVersion = 2
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 10
numChildren = 0
[zk: localhost:2181(CONNECTED) 1] rmr /accumulo/9354e975-7e2a-4207-8c7b-5d36c0e7765d/masters/goal_state
Node does not exist: /accumulo/9354e975-7e2a-4207-8c7b-5d36c0e7765d/masters/goal_state

I have run a 'stat' against all three servers and they seem properly structured with a leader and two followers. An md5sum of all zoo.cfg shows them to be identical.

The problem seems localized to the accumulo/935.... directory as I can create and delete znodes outside of that path fine but not inside of it.

For example:

[zk: localhost:2181(CONNECTED) 12] create /accumulo/9354e975-7e2a-4207-8c7b-5d36c0e7765d/fubar asdf
Node does not exist: /accumulo/9354e975-7e2a-4207-8c7b-5d36c0e7765d/fubar
[zk: localhost:2181(CONNECTED) 13] create /accumulo/fubar asdf
Created /accumulo/fubar
[zk: localhost:2181(CONNECTED) 14] ls /accumulo/fubar
[]
[zk: localhost:2181(CONNECTED) 15] rmr /accumulo/fubar
[zk: localhost:2181(CONNECTED) 16]

Here is my zoo.cfg:
tickTime=2000
initLimit=10
syncLimit=15
dataDir=/data/extera/zkeeper/data
clientPort=2181

maxClientCnxns=300
autopurge.snapRetainCount=10
autopurge.purgeInterval=1

server.1=cdf61:2888:3888
server.2=cdf62:2888:3888
server.3=cdf63:2888:3888

9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
4 years, 9 weeks, 2 days ago 0|i26xxb:
ZooKeeper ZOOKEEPER-2144

Provide a way to update the auth info on a connection

Improvement Open Major Unresolved Unassigned Karol Dudzinski Karol Dudzinski 16/Mar/15 06:48   16/Mar/15 19:51           0 2   The current auth info implementation makes it very difficult to work with expiring auth info. If a client fails over between servers, it resends its list of auth info in a FIFO order. Therefore, if any of the info has expired, it'll cause the session to be lost. There is currently no way to update or remove any existing info, only add. Any objections to adding an update or remove auth info method?

An alternate solution would be for ClientCnxn.AuthData to implement an equals method that only checks the scheme field. As the AuthData is stored in a set, this would have the same effect as an update operation. However, I'm not sure if there is a use case for supplying multiple bits of AuthData for the same scheme?
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
5 years, 1 week, 3 days ago 0|i26svb:
ZooKeeper ZOOKEEPER-2143

ZOOKEEPER-1525 Pass the operation and path to the AuthenticationProvider

Sub-task Resolved Major Implemented Unassigned Karol Dudzinski Karol Dudzinski 16/Mar/15 06:38   06/Oct/16 09:30 06/Oct/16 09:30         0 3   Currently, the AuthenticationProvider only gets passed the id of the client and the acl expression. If one wishes to perform auth checks based on the action or path being acted on, that needs to be included in the acl expression. This results in lots of potentially individual acl's being created which led us to find ZOOKEEPER-2141. It would be great if both the action and path were passed to the AuthenticationProvider.

I understand that this needs to be completely backwards compatible. One solution that comes to mind is to create an interface which extends AuthenticationProvider but adds a new matches which takes the additional parameters. Internally, ZK would use the new interface everywhere. To preserve compatibility, ProviderRegistry could check for classes implementing the original AuthenticationProvdier interface and wrap them to allow the new interface to be used everywhere internally. Any thoughts on this approach? Happy to provide a patch to demonstrate what I mean.
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
3 years, 24 weeks ago 0|i26suv:
ZooKeeper ZOOKEEPER-2142

JMX ObjectName is incorrect for observers

Bug Closed Trivial Fixed Edward Ribeiro Karol Dudzinski Karol Dudzinski 16/Mar/15 06:30   21/Jul/16 16:18 31/Oct/15 18:22 3.4.6, 3.5.1 3.4.7, 3.5.2, 3.6.0     0 8   Observers show up in JMX as StandaloneServer rather than Observer. 9223372036854775807 No Perforce job exists for this issue. 2 9223372036854775807
4 years, 20 weeks, 1 day ago 0|i26suf:
ZooKeeper ZOOKEEPER-2141

ACL cache in DataTree never removes entries

Bug Closed Blocker Fixed Adam Milne-Smith Karol Dudzinski Karol Dudzinski 16/Mar/15 06:28   21/Jul/16 16:18 06/Apr/16 14:09 3.4.6 3.4.9, 3.5.2     0 9   The problem and potential solutions are discussed in http://mail-archives.apache.org/mod_mbox/zookeeper-user/201502.mbox/browser

I will attach a proposed patch in due course.
9223372036854775807 No Perforce job exists for this issue. 8 9223372036854775807
3 years, 50 weeks, 1 day ago 0|i26su7:
ZooKeeper ZOOKEEPER-2140

NettyServerCnxn and NIOServerCnxn code should be improved

Improvement Resolved Major Fixed Mohammad Arshad Mohammad Arshad Mohammad Arshad 13/Mar/15 03:03   08/Aug/16 10:29 29/Jun/15 11:35   3.5.1, 3.6.0     0 5   Classes org.apache.zookeeper.server.NIOServerCnxn and org.apache.zookeeper.server.NettyServerCnxn have following need and scope for improvement
1) Duplicate code.
These two classes have around 250 line duplicate code. All the command code is duplicated
2) Many improvement/bugFix done in one class but not done in other class. These changes should be synced
For example

In NettyServerCnxn
{code}
// clone should be faster than iteration
// ie give up the cnxns lock faster
AbstractSet<ServerCnxn> cnxns;
synchronized (factory.cnxns) {
cnxns = new HashSet<ServerCnxn>(factory.cnxns);
}
for (ServerCnxn c : cnxns) {
c.dumpConnectionInfo(pw, false);
pw.println();
}
{code}
In NIOServerCnxn
{code}
for (ServerCnxn c : factory.cnxns) {
c.dumpConnectionInfo(pw, false);
pw.println();
}
{code}
3) NettyServerCnxn and NIOServerCnxn classes are bulky unnecessarily. Command classes have altogether different functionality, the command classes should go in different class files.
If this done it will be easy to add new command with minimal change to existing classes.
9223372036854775807 No Perforce job exists for this issue. 4 9223372036854775807
4 years, 38 weeks, 2 days ago 0|i26q87:
ZooKeeper ZOOKEEPER-2139

Support multiple ZooKeeper client, with different configurations, in a single JVM

Improvement Closed Blocker Fixed Mohammad Arshad Surendra Singh Lilhore Surendra Singh Lilhore 13/Mar/15 00:28   28/Oct/19 23:59 02/May/16 12:18 3.5.0 3.5.2, 3.6.0 java client   1 26   I have two ZK client in one JVM, one is secure client and second is normal client (For non secure cluster).

"zookeeper.sasl.client" system property is "true" by default, because of this my second client connection is failing.

We should pass all client configurations in client constructor like HDFS client.

For example :
{code}
public ZooKeeper(String connectString, int sessionTimeout, Watcher watcher, Configuration conf) throws IOException
{
......
......
}
{code}
9223372036854775807 No Perforce job exists for this issue. 14 9223372036854775807
3 years, 9 weeks, 1 day ago 0|i26q3j:
ZooKeeper ZOOKEEPER-2138

ZOOKEEPER-2135 ZooKeeper C client testing is failing

Sub-task Resolved Major Duplicate Michi Mutsuzaki Hongchao Deng Hongchao Deng 11/Mar/15 17:00   19/Dec/19 18:01 25/Mar/15 13:26         0 3   After testHammer has been fixed, now test-core-java is successful and test-core-cppunit emerges to fail:

https://builds.apache.org/view/S-Z/view/ZooKeeper/job/ZooKeeper-trunk/2624/console
https://builds.apache.org/view/S-Z/view/ZooKeeper/job/PreCommit-ZOOKEEPER-Build/2557/console

I have try git bitsect under "src/c" and figure out the JIRA causing problem:

* ZOOKEEPER-2114 (might be some other jira between ZK-2114 and ZK-2049)
My local jenkins showed the error messages:
{code}
[exec] /var/lib/jenkins/workspace/zk-trunk/src/c/tests/TestClient.cc:1072: Assertion: assertion failed [Expression: ctx.waitForConnected(zk)]
{code}
sky 9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
5 years, 1 day ago 0|i26npb:
ZooKeeper ZOOKEEPER-2137

ZOOKEEPER-2135 Make testPortChange() less flaky

Sub-task Closed Major Fixed Michael Han Hongchao Deng Hongchao Deng 10/Mar/15 18:05   21/Jul/16 16:18 15/Jun/16 13:16 3.5.0 3.5.2, 3.6.0 tests   0 8   The cause of flaky failure of testPortChange() is a race in sync().

I figured out it could take some time to fix sync(). Meanwhile, we can make testPortChange() less flaky by doing reconfig on the leader. We can change this back in the fix of ZOOKEEPER-2136.
flakey 9223372036854775807 No Perforce job exists for this issue. 5 9223372036854775807
3 years, 40 weeks, 1 day ago
Reviewed
0|i26lrj:
ZooKeeper ZOOKEEPER-2136

ZOOKEEPER-2135 Sync() should get quorum acks.

Sub-task Open Major Unresolved Flavio Paiva Junqueira Hongchao Deng Hongchao Deng 10/Mar/15 01:27   14/Dec/19 06:06   3.5.5 3.7.0     1 10   Currently if the sync packet goes to leader it doesn't get quorum acks. This is a problem during reconfig and leader changes. testPortChange() flaky failure is caused by such case.

I proposed to change sync() semantics to require quorum acks in any case.

9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
38 weeks, 1 day ago 0|i26k3b:
ZooKeeper ZOOKEEPER-2135

fix trunk build

Bug Open Major Unresolved Michael Han Michi Mutsuzaki Michi Mutsuzaki 08/Mar/15 17:16   14/Dec/19 06:07     3.7.0     0 2   ZOOKEEPER-2000, ZOOKEEPER-2080, ZOOKEEPER-2134, ZOOKEEPER-2136, ZOOKEEPER-2137, ZOOKEEPER-2138, ZOOKEEPER-2152, ZOOKEEPER-2637 9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
5 years, 2 weeks, 4 days ago 0|i26htz:
ZooKeeper ZOOKEEPER-2134

ZOOKEEPER-2135 AsyncHammerTest.testHammer fails intermittently

Sub-task Resolved Blocker Fixed Michi Mutsuzaki Michi Mutsuzaki Michi Mutsuzaki 08/Mar/15 17:14   09/Mar/15 17:35 09/Mar/15 14:26   3.5.1, 3.6.0 tests   0 4   The trunk build has been red for a while because of this (and ZOOKEEPER-2000 and ZOOKEEPER-2080). We should fix this sooner than later. 9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
5 years, 2 weeks, 3 days ago 0|i26htr:
ZooKeeper ZOOKEEPER-2133

zkperl: Segmentation fault if getting a node with null value

Bug Closed Major Fixed Botond Hejj Botond Hejj Botond Hejj 05/Mar/15 11:49   21/Jul/16 16:18 11/Mar/16 01:36 3.4.6, 3.5.0 3.4.9, 3.5.2, 3.6.0 contrib-bindings   0 2   If Node content is null:

[zk: (CONNECTED) 0] get /apps
null
cZxid = 0x10000000d

than

my $data = $zk->{zkh}->get('/apps');

causing a core dump with Segmentation fault
9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
4 years, 1 week, 6 days ago
Reviewed
zkperl 0|i26eqv:
ZooKeeper ZOOKEEPER-2132

Follower::syncWithLeader should roll logs before taking snapshot

Bug Open Major Unresolved Unassigned Asad Saeed Asad Saeed 26/Feb/15 13:49   04/Feb/16 07:17   3.4.6   server   0 4   If multiple leader elections occur before SyncRequestProcessor takes a snapshot and rolls logs (at least 50000 transactions by default). PurgeTxnLog may inadvertently delete the current transaction log file.

Follower::syncWithLeader currently takes a snapshot after it is synced with the leader but does not roll logs. If a zookeeper restart of a quorum of nodes occurs, the cluster may silently revert back to the last snapshot, loosing all transactions in the log!
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
4 years, 7 weeks ago 0|i263vr:
ZooKeeper ZOOKEEPER-2131

Error:KeeperErrorCode = NoNode

Bug Open Major Unresolved Unassigned dirtdiver512 dirtdiver512 24/Feb/15 12:25   24/Feb/15 12:25   3.5.0       0 1   ubuntu testing with kafka getting error below:

[2015-02-24 11:11:25,226] INFO Got user-level KeeperException when processing sessionid:0x14bbc922a2b0002 type:create cxid:0x1a zxid:0xe8 txntype:-1 reqpath:n/a Error Path:/consumers/console-consumer-67319/owners Error:KeeperErrorCode = NoNode for /consumers/console-consumer-67319/owners (org.apache.zookeeper.server.PrepRequestProcessor)
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
5 years, 4 weeks, 2 days ago 0|i25zdr:
ZooKeeper ZOOKEEPER-2130

Command to get ensemble summary

New Feature Open Minor Unresolved Unassigned Mohammad Arshad Mohammad Arshad 24/Feb/15 05:49   14/Dec/19 06:06     3.7.0     0 1   It is good to have a command which can give complete summary of zookeeper ensemble. Ensemble summary should give information about who is leader, which are followers, observers.

Consider a zookeeper cluster with following configurations
server.1=localhost:33230:33235:participant;localhost:33222
server.2=localhost:33231:33236:participant;localhost:33223
server.3=localhost:33232:33237:participant;localhost:33224
server.4=localhost:33233:33238:participant;localhost:33225
server.5=localhost:33234:33239:participant;localhost:33226

When four servers are running and we execute esum(Ensemble Summary Command) command we should get status of all the servers and their roles

Example:
{quote}
server.1=localhost:33230:33235:participant;localhost:33222 {color:green}FOLLOWING{color}
Client Connections:1

server.2=localhost:33231:33236:participant;localhost:33223 {color:green}FOLLOWING{color}
Client Connections:0

server.3=localhost:33232:33237:participant;localhost:33224 {color:red}NOT RUNNING{color}

server.4=localhost:33233:33238:participant;localhost:33225 {color:green}FOLLOWING{color}
Client Connections:0

server.5=localhost:33234:33239:participant;localhost:33226 {color:blue}LEADING{color}
Client Connections:0
{quote}

9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
5 years, 1 week, 2 days ago 0|i25yyf:
ZooKeeper ZOOKEEPER-2129

ruok command is not consistent with other four letter commands

Bug Resolved Trivial Not A Problem Unassigned Mohammad Arshad Mohammad Arshad 24/Feb/15 05:15   19/Dec/19 17:59 05/Mar/15 05:30         0 2   ruok commands prints the output in the same line unlike the other four letter commands which print output in next line. Even though output is correct it is difficult to notice the output specially for a first time user. Its output should contain new line character as other four letter command's output

ruok command output:
{code}
HOST1:/home # echo ruok | netcat 10.x.x.x 2181
imokHOST1:/home #
{code}

conf Command output:
{code}
HOST1:/home # echo conf | netcat 10.x.x.x 2181
clientPort=2181
dataDir=/tmp/zookeeper/data/version-2
dataLogDir=/tmp/zookeeper/data/version-2
tickTime=2000
.......
HOST1:/home #
{code}
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
5 years, 3 weeks ago 0|i25ywn:
ZooKeeper ZOOKEEPER-2128

zoo_aremove_watchers API is incorrect

Bug Open Major Unresolved Dave Gosselin Dave Gosselin Dave Gosselin 23/Feb/15 22:46   05/Feb/20 07:16   3.6.0 3.7.0, 3.5.8     0 4   The C API for zoo_aremove_watchers incorrectly specifies the seventh argument as a pointer to a function pointer. It should be simply a function pointer only. 9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
8 weeks, 3 days ago 0|i25yhz:
ZooKeeper ZOOKEEPER-2127

Document zkCli.sh

Bug Resolved Major Done Unassigned Joe Gamache Joe Gamache 23/Feb/15 06:48   04/Aug/19 08:45 04/Aug/19 08:45         0 2   According to the answer provided to this stack overflow question:

http://stackoverflow.com/questions/28589703/zookeeper-zkcli-sh-create-switches-documentation/28594057#28594057

the zkCli.sh script is not documented in terms of what all the switches mean. Such documentation should be provided.
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
32 weeks, 4 days ago 0|i25x4v:
ZooKeeper ZOOKEEPER-2126

Improve exit log messsage of EventThread and SendThread by adding SessionId

Improvement Resolved Major Fixed Surendra Singh Lilhore Zhihai Xu Zhihai Xu 21/Feb/15 18:58   15/May/15 07:27 15/May/15 00:16 3.6.0 3.4.7, 3.5.1, 3.6.0 java client   0 5   We saw the following out of order log when close Zookeeper client session.
{code}
2015-02-16 06:01:12,985 INFO org.apache.zookeeper.ZooKeeper: Session: 0x24b8df4044005d4 closed
.....................................
2015-02-16 06:01:12,995 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down
{code}

This logs are very confusing if a new Zookeeper client session is created between these two logs. We may think new Zookeeper client session shutdown it EventThread instead of the old closed Zookeeper client session.

Should we wait for sendThread and eventThread died in the ClientCnxn.close?
We can add the following code in ClientCnxn.close.
{code}
sendThread.join(timeout);
eventThread.join(timeout);
{code}
with the change, we won't interleave old closed session with new session.
We can also create a new close API to support this so we won't affect the old code if people use old close API.
9223372036854775807 No Perforce job exists for this issue. 3 9223372036854775807
4 years, 44 weeks, 6 days ago 0|i25waf:
ZooKeeper ZOOKEEPER-2125

ZOOKEEPER-2120 SSL on Netty client-server communication

Sub-task Resolved Major Fixed Hongchao Deng Hongchao Deng Hongchao Deng 20/Feb/15 17:32   25/Jul/19 02:42 17/Mar/15 13:21   3.5.1, 3.6.0     0 20   Supporting SSL on Netty client-server communication.
1. It supports keystore and trustore usage.
2. It adds an additional ZK server port which supports SSL. This would be useful for rolling upgrade.

RB: https://reviews.apache.org/r/31277/

The patch includes three files:
* testing purpose keystore and truststore under "$(ZK_REPO_HOME)/src/java/test/data/ssl". Might need to create "ssl/".
* latest ZOOKEEPER-2125.patch

h2. How to use it
You need to set some parameters on both ZK server and client.

h3. Server
You need to specify a listening SSL port in "zoo.cfg":
{code}
secureClientPort=2281
{code}
Just like what you did with "clientPort". And then set some jvm flags:
{code}
export SERVER_JVMFLAGS="-Dzookeeper.serverCnxnFactory=org.apache.zookeeper.server.NettyServerCnxnFactory -Dzookeeper.ssl.keyStore.location=/root/zookeeper/ssl/testKeyStore.jks -Dzookeeper.ssl.keyStore.password=testpass -Dzookeeper.ssl.trustStore.location=/root/zookeeper/ssl/testTrustStore.jks -Dzookeeper.ssl.trustStore.password=testpass"
{code}
Please change keystore and truststore parameters accordingly.

h3. Client
You need to set jvm flags:
{code}
export CLIENT_JVMFLAGS="-Dzookeeper.clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty -Dzookeeper.client.secure=true -Dzookeeper.ssl.keyStore.location=/root/zookeeper/ssl/testKeyStore.jks -Dzookeeper.ssl.keyStore.password=testpass -Dzookeeper.ssl.trustStore.location=/root/zookeeper/ssl/testTrustStore.jks -Dzookeeper.ssl.trustStore.password=testpass"
{code}
change keystore and truststore parameters accordingly.
And then connect to the server's SSL port, in this case:
{code}
bin/zkCli.sh -server 127.0.0.1:2281
{code}

If you have any feedback, you are more than welcome to discuss it here!
ssl-tls 9223372036854775807 No Perforce job exists for this issue. 21 9223372036854775807
1 year, 6 weeks, 1 day ago 0|i25vkv:
ZooKeeper ZOOKEEPER-2124

Allow Zookeeper version string to have underscore '_'

Bug Resolved Major Fixed Chris Nauroth Jerry He Jerry He 20/Feb/15 13:58   14/Sep/15 16:35 24/May/15 02:38 3.4.6 3.4.7, 3.5.1, 3.6.0     0 7   Using Bigtop or other RPM build for Zookeeper, there is a problem with using the hyphen '-' character in the version string:
{noformat}
[bigdata@bdvs1166 bigtop]$ gradle zookeeper-rpm
:buildSrc:compileJava UP-TO-DATE
:buildSrc:compileGroovy UP-TO-DATE
:buildSrc:processResources UP-TO-DATE
:buildSrc:classes UP-TO-DATE
:buildSrc:jar UP-TO-DATE
:buildSrc:assemble UP-TO-DATE
:buildSrc:compileTestJava UP-TO-DATE
:buildSrc:compileTestGroovy UP-TO-DATE
:buildSrc:processTestResources UP-TO-DATE
:buildSrc:testClasses UP-TO-DATE
:buildSrc:test UP-TO-DATE
:buildSrc:check UP-TO-DATE
:buildSrc:build UP-TO-DATE
:zookeeper_vardefines
:zookeeper-download
:zookeeper-tar
Copy /home/bigdata/bigtop/dl/zookeeper-3.4.6-IBM-1.tar.gz to /home/bigdata/bigtop/build/zookeeper/tar/zookeeper-3.4.6-IBM-1.tar.gz
:zookeeper-srpm
error: line 64: Illegal char '-' in: Version: 3.4.6-IBM-1
:zookeeper-srpm FAILED

FAILURE: Build failed with an exception.

* Where:
Script '/home/bigdata/bigtop/packages.gradle' line: 462

* What went wrong:
Execution failed for task ':zookeeper-srpm'.
> Process 'command 'rpmbuild'' finished with non-zero exit value 1

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.

BUILD FAILED
{noformat}

Also, according to the [rpm-maven-plugin|http://mojo.codehaus.org/rpm-maven-plugin/ident-params.html] documentation:
{noformat}

version
The version number to use for the RPM package. By default, this is the project version. This value cannot contain a dash (-) due to contraints in the RPM file naming convention. Any specified value will be truncated at the first dash

release
The release number of the RPM.
Beginning with release 2.0-beta-2, this is an optional parameter. By default, the release will be generated from the modifier portion of the project version using the following rules:
If no modifier exists, the release will be 1.
If the modifier ends with SNAPSHOT, the timestamp (in UTC) of the build will be appended to end.
All instances of '-' in the modifier will be replaced with '_'.
If a modifier exists and does not end with SNAPSHOT, "_1" will be appended to end.

{noformat}

We should allow underscore '_' as part of the version string. e.g.
3.4.6_abc_1
9223372036854775807 No Perforce job exists for this issue. 3 9223372036854775807
4 years, 43 weeks, 4 days ago 0|i25v9r:
ZooKeeper ZOOKEEPER-2123

ZOOKEEPER-2120 Provide implementation of X509 AuthenticationProvider

Sub-task Resolved Minor Fixed Ian Dimayuga Hongchao Deng Hongchao Deng 18/Feb/15 18:02   29/Mar/15 07:24 28/Mar/15 15:13   3.5.1, 3.6.0     0 7   9223372036854775807 No Perforce job exists for this issue. 6 9223372036854775807
4 years, 51 weeks, 4 days ago 0|i25scv:
ZooKeeper ZOOKEEPER-2122

ZOOKEEPER-2063 Impplement SSL support in the Zookeeper C client library

Sub-task Resolved Trivial Fixed Mate Szalay-Beko Ashish Amarnath Ashish Amarnath 18/Feb/15 12:24   26/Nov/19 04:26 21/Nov/19 03:54 3.5.0 3.6.0 c client   5 12 0 51600   Implement SSL support in the Zookeeper C client library to work with the secure server.
100% 100% 51600 0 build, pull-request-available, security, ssl-tls 9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
17 weeks ago
Incompatible change, Reviewed
0|i25rsv:
ZooKeeper ZOOKEEPER-2121

Documentation of create method is unclear, incorrect, or both

Bug Open Major Unresolved Unassigned Joe Gamache Joe Gamache 18/Feb/15 12:08   16/Aug/15 04:04   3.4.6   documentation   0 3   If you go to the create method documentation here:
http://zookeeper.apache.org/doc/r3.4.6/api/index.html

Then you see:
{code}
public String create(String path,
byte[] data,
List<ACL> acl,
CreateMode createMode)
throws KeeperException,
InterruptedException
Create a node with the given path. The node data will be the given data, and node acl will be the given acl.
The flags argument specifies whether the created node will be ephemeral or not.

An ephemeral node will be removed by the ZooKeeper automatically when the session associated with the creation of the node expires.

The flags argument can also specify to create a sequential node. The actual path name of a sequential node will be the given path plus a suffix "i" where i is the current sequential number of the node. The sequence number is always fixed length of 10 digits, 0 padded. Once such a node is created, the sequential number will be incremented by one.
{code}

While there are 'path', 'data', 'acl', and 'createMode' arguments, there is no "flags argument". This documentation needs to be corrected to be clear, unambiguous, and perhaps provide and example.
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
4 years, 31 weeks, 4 days ago 0|i25rrj:
ZooKeeper ZOOKEEPER-2120

SSL feature on Netty

New Feature Open Major Unresolved Hongchao Deng Hongchao Deng Hongchao Deng 17/Feb/15 15:20   13/May/15 13:08           0 3   ZOOKEEPER-2119, ZOOKEEPER-2123, ZOOKEEPER-2125, ZOOKEEPER-2148, ZOOKEEPER-2153 As we discussed in ZOOKEEPER-2094, the SSL work would be divided into several subtask:

1. Provide implementation of X509 AuthenticationProvider
2. Modify ZooKeeper Netty server and client to support SSL
3. Modify ZooKeeperServerMain to support SSL

This is the umbrella task.
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
5 years, 5 weeks, 2 days ago 0|i25qlj:
ZooKeeper ZOOKEEPER-2119

ZOOKEEPER-2120 Netty client docs

Sub-task Resolved Major Fixed Hongchao Deng Hongchao Deng Hongchao Deng 17/Feb/15 11:42   25/Feb/15 06:15 24/Feb/15 12:47   3.5.1, 3.6.0     0 5   ZOOKEEPER-2069 adds Netty client option. We need to add docs on how to use it. 9223372036854775807 No Perforce job exists for this issue. 5 9223372036854775807
5 years, 4 weeks, 1 day ago 0|i25q4f:
ZooKeeper ZOOKEEPER-2118

CLONE - Netty for quorum communication

Bug Open Major Unresolved Unassigned ysl871308 ysl871308 15/Feb/15 01:32   01/Apr/15 01:52       quorum   0 2   We need Netty in quorum communication to make use of SSL/auth feature in Netty.

This might need more thoughts like ZOOKEEPER-901. This issue would be a good place to start discussing thoughts.
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
5 years, 5 weeks, 3 days ago 0|i25nef:
ZooKeeper ZOOKEEPER-2117

"caught end of stream", server: "Stale state" of a Zk client just after connecting

Bug Open Critical Unresolved Unassigned Bruno Gauthier Bruno Gauthier 13/Feb/15 10:13   03/Mar/16 00:59   3.5.0   c client   0 2   Windows 8.1, Windows 2012, Visual Studio 2012 Hi All,

Under WIndows 8.1 and 2012, using the ZooKeeper C client 3.5.0, when running my ZooKeeper client, just after the ZooKeeper Client is connecting with the ZooKeeper server, the ZooKeeper server is generating a “caught end of stream” exception and deciding my is ZooKeeper client is not responsive: Zookeeper.c::check_events, line 2298: ESTALE. (see log below).

This problem systematically appears if the ZooKeeper DLL is NOT link statically with the Visual Studio debug version of the threaded runtime library.
This is reproducible 10/10

In clear, Windows ZooKeeper C client will works only if you link your ZooKeeper DLL with the switch "/MTd" (see VS Studio->Project->Configuration properties->C/C++->Code generation->runtime library)

Thanks

Bruno

========================================
ZooKeeper server log
========================================

2015-02-06 13:19:57,552 [myid:vgcclustermgr] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:31000:NIOServerCnxnFactory@197] - Accepted socket connection from /10.1.200.237:63499
2015-02-06 13:19:57,553 [myid:vgcclustermgr] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:31000:ZooKeeperServer@868] - Client attempting to establish new session at /10.1.200.237:63499
2015-02-06 13:19:57,554 [myid:vgcclustermgr] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:31000:NIOServerCnxnFactory@197] - Accepted socket connection from /10.1.200.237:63500
2015-02-06 13:19:57,554 [myid:vgcclustermgr] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:31000:ZooKeeperServer@868] - Client attempting to establish new session at /10.1.200.237:63500
2015-02-06 13:19:57,555 [myid:vgcclustermgr] - INFO [SyncThread:0:ZooKeeperServer@617] - Established session 0x14b5bfcba7b0409 with negotiated timeout 80000 for client /10.1.200.237:63499
2015-02-06 13:19:57,555 [myid:vgcclustermgr] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:31000:NIOServerCnxn@357] - caught end of stream exception
EndOfStreamException: Unable to read additional data from client sessionid 0x14b5bfcba7b0409, likely client has closed socket
at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
at java.lang.Thread.run(Thread.java:744)
2015-02-06 13:19:57,555 [myid:vgcclustermgr] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:31000:NIOServerCnxn@1007] - Closed socket connection for client /10.1.200.237:63499 which had sessionid 0x14b5bfcba7b0409
2015-02-06 13:19:57,559 [myid:vgcclustermgr] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:31000:NIOServerCnxnFactory@197] - Accepted socket connection from /10.1.200.237:63501
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
4 years, 3 weeks ago 0|i25lpr:
ZooKeeper ZOOKEEPER-2116

zkCli.sh doesn't honor host:port parameter

Bug Resolved Critical Implemented Surendra Singh Lilhore Maxim Novikov Maxim Novikov 11/Feb/15 18:48   28/May/15 23:51 28/May/15 05:50 3.4.6 3.6.0 java client, scripts   1 4   Ubuntu 12 This doc http://zookeeper.apache.org/doc/r3.1.2/zookeeperStarted.html ("Connecting to ZooKeeper" section) says:

Once ZooKeeper is running, you have several options for connection to it:

Java: Use
bin/zkCli.sh 127.0.0.1:2181

In fact, it doesn't work that way. I am running ZooKeeper with a different port to listen to client connections (2888), and this command

{code}
bin/zkCli.sh 127.0.0.1:2888
{code}

is still trying to connect to 2181.

{code:title=output|borderStyle=solid}
Connecting to localhost:2181
2015-02-11 15:38:14,415 [myid:] - INFO [main:Environment@100] - Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT
2015-02-11 15:38:14,421 [myid:] - INFO [main:Environment@100] - Client environment:host.name=localhost
2015-02-11 15:38:14,421 [myid:] - INFO [main:Environment@100] - Client environment:java.version=1.7.0_17
2015-02-11 15:38:14,424 [myid:] - INFO [main:Environment@100] - Client environment:java.vendor=Oracle Corporation
2015-02-11 15:38:14,424 [myid:] - INFO [main:Environment@100] - Client environment:java.home=/usr/java/jdk1.7.0_17/jre
2015-02-11 15:38:14,424 [myid:] - INFO [main:Environment@100] - Client environment:java.class.path=/opt/zookeeper-3.4.6/bin/../build/classes:/opt/zookeeper-3.4.6/bin/../build/lib/*.jar:/opt/zookeeper-3.4.6/bin/../lib/slf4j-log4j12-1.6.1.jar:/opt/zookeeper-3.4.6/bin/../lib/slf4j-api-1.6.1.jar:/opt/zookeeper-3.4.6/bin/../lib/netty-3.7.0.Final.jar:/opt/zookeeper-3.4.6/bin/../lib/log4j-1.2.16.jar:/opt/zookeeper-3.4.6/bin/../lib/jline-0.9.94.jar:/opt/zookeeper-3.4.6/bin/../zookeeper-3.4.6.jar:/opt/zookeeper-3.4.6/bin/../src/java/lib/*.jar:../conf::/usr/share/antlr3/lib/antlr-3.5-complete-no-st3.jar
2015-02-11 15:38:14,425 [myid:] - INFO [main:Environment@100] - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2015-02-11 15:38:14,425 [myid:] - INFO [main:Environment@100] - Client environment:java.io.tmpdir=/tmp
2015-02-11 15:38:14,425 [myid:] - INFO [main:Environment@100] - Client environment:java.compiler=<NA>
2015-02-11 15:38:14,425 [myid:] - INFO [main:Environment@100] - Client environment:os.name=Linux
2015-02-11 15:38:14,425 [myid:] - INFO [main:Environment@100] - Client environment:os.arch=amd64
2015-02-11 15:38:14,426 [myid:] - INFO [main:Environment@100] - Client environment:os.version=3.8.0-41-generic
2015-02-11 15:38:14,426 [myid:] - INFO [main:Environment@100] - Client environment:user.name=mnovikov
2015-02-11 15:38:14,426 [myid:] - INFO [main:Environment@100] - Client environment:user.home=/home/mnovikov
2015-02-11 15:38:14,426 [myid:] - INFO [main:Environment@100] - Client environment:user.dir=/opt/zookeeper-3.4.6/bin
2015-02-11 15:38:14,428 [myid:] - INFO [main:ZooKeeper@438] - Initiating client connection, connectString=localhost:2181 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@3107eafc
Welcome to ZooKeeper!
2015-02-11 15:38:14,471 [myid:] - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread@975] - Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
2015-02-11 15:38:14,479 [myid:] - WARN [main-SendThread(localhost:2181):ClientCnxn$SendThread@1102] - Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:692)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
{code}

PS1 I can connect to ZK at 2888 using ZK Java client from code specifying the correct port with no issues. But CLI seems just to ignore the provided host:port parameter.

PS2 Tried to run it with the pre-defined ZOOCFGDIR environment variable (to point to the path with the config file where the client port is set to 2888). No luck, same results as shown above.
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
4 years, 42 weeks, 6 days ago 0|i25ihr:
ZooKeeper ZOOKEEPER-2115

Initialize command succeeds even though it didn't in case of permission errors on the data directory

Bug Patch Available Trivial Unresolved Biju Nair Manikandan Narayanaswamy Manikandan Narayanaswamy 05/Feb/15 19:44   14/Dec/19 06:06   3.5.0 3.7.0 scripts   0 4   while testing single user mode that the initialize command succeeds even though it didn't in case of permission errors on the data directory:
{code}
....
+ exec /usr/lib/zookeeper/bin/zkServer-initialize.sh --myid=1
mkdir: cannot create directory `/var/lib/zookeeper/version-2': Permission denied
mkdir: cannot create directory `/var/lib/zookeeper/version-2': Permission denied
/usr/lib/zookeeper/bin/zkServer-initialize.sh: line 112: /var/lib/zookeeper/myid: Permission denied
{code}
9223372036854775807 No Perforce job exists for this issue. 5 9223372036854775807
4 years, 2 weeks, 5 days ago 0|i259nj:
ZooKeeper ZOOKEEPER-2114

jute generated allocate_* functions are not externally visible

Bug Resolved Major Fixed Tim Crowder Tim Crowder Tim Crowder 05/Feb/15 03:06   22/Feb/15 17:52 22/Feb/15 16:12 3.5.0 3.4.7, 3.5.1, 3.6.0 c client   0 4   Some jute generated functions (e.g. allocate_ACL_vector) that should be publicly exported are given local (vs global) linkage. This is due to an incomplete regex for EXPORT_SYMBOLS in the C Makefile.am.

Without allocate_ACL_vector it's not possible to set ACL lists from C.
The regex should include "allocate_" :

EXPORT_SYMBOLS = '(zoo_|zookeeper_|zhandle|Z|format_log_message|log_message|logLevel|deallocate_|allocate_|zerror|is_unrecoverable)'
9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
5 years, 4 weeks, 4 days ago Expose jute-generated allocate_XXX functions in libzookeeper. 0|i2582v:
ZooKeeper ZOOKEEPER-2113

SSL support for ClientCnxnSocketNetty

Bug Open Major Unresolved Hongchao Deng Hongchao Deng Hongchao Deng 29/Jan/15 16:38   29/Jan/15 16:55           0 2   This is a SSL feature on netty client. 9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
5 years, 8 weeks ago 0|i24z4v:
ZooKeeper ZOOKEEPER-2112

Netty for quorum communication

Bug Open Major Unresolved Unassigned Hongchao Deng Hongchao Deng 27/Jan/15 20:00   01/Apr/15 01:52       quorum   0 2   Add Netty option to replace NIO for quorum communication. 9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
5 years, 8 weeks, 1 day ago 0|i24vlr:
ZooKeeper ZOOKEEPER-2111

Not isAlive states should be synchronized in ClientCnxn

Bug Resolved Major Fixed Hongchao Deng Hongchao Deng Hongchao Deng 27/Jan/15 14:21   21/Feb/20 21:55 31/Jan/15 02:11   3.5.1, 3.6.0 java client   0 5   In ClientCnxn.queuePacket, it checks variables of state and closing and then make decisions. There is toctou race in queuePacket():
{code}
if (!state.isAlive() || closing) {
conLossPacket(packet);
} else {
...
}
{code}

A possible race:
in SendThread.run():
{code}
while (state.isAlive()) {
...
}
cleanup();
{code}

When it checks in queuePacket(), state is still alive. Then state isn't alive, SendThread.run() cleans up outgoingQueue. Then queuePacket adds packet to outgoingQueue. The packet should be waken up with exception. But it won't at this case.
9223372036854775807 No Perforce job exists for this issue. 4 9223372036854775807
5 years, 7 weeks, 5 days ago 0|i24uvb:
ZooKeeper ZOOKEEPER-2110

Typo fixes in the ZK documentation

Improvement Resolved Minor Fixed Jeffrey Schroeder Jeffrey Schroeder Jeffrey Schroeder 26/Jan/15 23:11   29/Jan/15 21:29 27/Jan/15 11:33 3.5.0 3.5.1, 3.6.0 documentation   0 3   So as part of building an Aurora cluster ontop of Mesos, I wanted to learn ZK. I spent an evening reading much of the ZK documentation and noticed many grammatical or spelling errors. Being a good OSS citizen, I've went through the effort to fix them and create a patch fixing it. 9223372036854775807 No Perforce job exists for this issue. 2 9223372036854775807
5 years, 8 weeks, 1 day ago 0|i24ttr:
ZooKeeper ZOOKEEPER-2109

Typo in src/c/src/load_gen.c

Bug Resolved Trivial Fixed Surendra Singh Lilhore Emmanuel Bourg Emmanuel Bourg 26/Jan/15 07:29   16/Mar/15 07:24 16/Mar/15 01:28 3.5.0 3.5.1, 3.6.0     0 5   There is a minor typo in {{src/c/src/load_gen.c}}, "Succesfully" should be spelled "Successfully" 9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
5 years, 1 week, 3 days ago 0|i24spb:
ZooKeeper ZOOKEEPER-2108

Compilation error in ZkAdaptor.cc with GCC 4.7 or later

Bug Open Minor Unresolved Unassigned Emmanuel Bourg Emmanuel Bourg 26/Jan/15 07:19   11/Oct/19 11:02   3.4.6       1 1   Hi,

Debian and Fedora have a patch fixing a compilation failure in ZkAdaptor.cc but it doesn't appear to be fixed in the upcoming version 3.5.0. This issue is similar to ZOOKEEPER-470 and ZOOKEEPER-1795.

The error is :
{code}
g++ -DHAVE_CONFIG_H -I. -I.. -D_FORTIFY_SOURCE=2 -I/home/ebourg/packaging/zookeeper/src/contrib/zktreeutil/../../c/include -I/home/ebourg/packaging/zookeeper/src/contrib/zktreeutil/../../c/generated -I../include -I/usr/local/include -I/usr/include -I/usr/include/libxml2 -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -MT ZkAdaptor.o -MD -MP -MF .deps/ZkAdaptor.Tpo -c -o ZkAdaptor.o ZkAdaptor.cc
ZkAdaptor.cc: In member function ‘void zktreeutil::ZooKeeperAdapter::reconnect()’:
ZkAdaptor.cc:220:21: error: ‘sleep’ was not declared in this scope
sleep (1);
{code}

This is fixed by including unistd.h in ZkAdaptor.cc or ZkAdaptor.h

The Debian patch:
https://sources.debian.net/src/zookeeper/3.4.5%2Bdfsg-2/debian/patches/ftbfs-gcc-4.7.diff/

and the Fedora patch:
http://pkgs.fedoraproject.org/cgit/zookeeper.git/tree/zookeeper-3.4.5-zktreeutil-gcc.patch
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
22 weeks, 6 days ago 0|i24sov:
ZooKeeper ZOOKEEPER-2107

zookeeper client should support custom HostProviders

Improvement Resolved Major Fixed Robert Kamphuis Robert Kamphuis Robert Kamphuis 23/Jan/15 02:12   16/Mar/15 07:24 16/Mar/15 04:18 3.5.0 3.5.1, 3.6.0 java client   0 5   The zookeeper client currently contains a StaticHostProvider and no means to replace it with your own implementation of the existing HostProvider interface. It would be great if the zookeeper client would enable you to create an instance with your own implementation of the HostProvider interface.

We have been testing this change with our implementation of HostProvider which does the name to ip lookup at the time of finding the next() server to connect to. In our AWS based deployments, this enables that applications can actually connect to swapped out zookeeper-servers which typically get a new ip address. With the current StaticHostProvider in practice you will need to restart the application to see the replaced zookeeper as the application continues to try to connect to the old no longer existing IP address.

With this little change plus a straightforward implementation of the HostProvider interface (our LateResolvingHostProvider), we can replace the zookeeper servers one at a time, re-assigning the elastic ips we use for the zookeeper servers, and have the application servers re-connect to the zookeeper cluster including the replaced ones without any downtime and without having to rely on the elastic ips for client to zookeeper-server connections. The reason for not using elastic-ips in the connect-strings but use the names which map to the (changing) private ips is to be able to rely on security-groups to control access.

While this seems very specific for AWS, this anyway seems a generic improvement for other deployments where the mapping from server name to IP address is not static.
9223372036854775807 No Perforce job exists for this issue. 4 9223372036854775807
5 years, 1 week, 3 days ago 0|i24pnj:
ZooKeeper ZOOKEEPER-2106

Error when reading from leader causes JVM to hang

Bug Resolved Critical Invalid Unassigned Robert Joseph Evans Robert Joseph Evans 13/Jan/15 11:28   13/Jan/15 11:54 13/Jan/15 11:54 3.4.5       0 1   I tried looking through existing JIRA for something like this, but the closest I came was ZOOKEEPER-2104. It looks very similar, but I don't know if it really is the same thing. Essentially we had a 5 node ensemble for a large storm cluster. For a few of the nodes at the same time they get an error that looks like.

{code}
WARN [RecvWorker:2:QuorumCnxManager$RecvWorker@762] - Connection broken for id 2, my id = 4, error =
java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:392)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:747)
WARN [RecvWorker:2:QuorumCnxManager$RecvWorker@765] - Interrupting SendWorker
WARN [SendWorker:2:QuorumCnxManager$SendWorker@679] - Interrupted while waiting for message on queue
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:831)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:62)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:667)
WARN [SendWorker:2:QuorumCnxManager$SendWorker@688] - Send worker leaving thread
WARN [QuorumPeer[myid=4]/0.0.0.0:50512:Follower@89] - Exception when following the leader
java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:189)
at java.net.SocketInputStream.read(SocketInputStream.java:121)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at java.io.BufferedInputStream.read(BufferedInputStream.java:254)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63)
at org.apache.zookeeper.server.quorum.QuorumPacket.deserialize(QuorumPacket.java:83)
at org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:108)
at org.apache.zookeeper.server.quorum.Learner.readPacket(Learner.java:152)
at org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:85)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:740)
INFO [QuorumPeer[myid=4]/0.0.0.0:50512:Follower@166] - shutdown called
java.lang.Exception: shutdown Follower
at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:166)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:744)
{code}

After that all of the connections are shut down
{code}
INFO [QuorumPeer[myid=4]/0.0.0.0:50512:NIOServerCnxn@1001] - Closed socket connection for client ...
{code}

but it does not manage to have the JVM shut down

{code}
INFO [QuorumPeer[myid=4]/0.0.0.0:50512:FollowerZooKeeperServer@139] - Shutting down
INFO [QuorumPeer[myid=4]/0.0.0.0:50512:ZooKeeperServer@419] - shutting down
INFO [QuorumPeer[myid=4]/0.0.0.0:50512:FollowerRequestProcessor@105] - Shutting down
INFO [QuorumPeer[myid=4]/0.0.0.0:50512:CommitProcessor@181] - Shutting down
INFO [FollowerRequestProcessor:4:FollowerRequestProcessor@95] - FollowerRequestProcessor exited loop!
INFO [QuorumPeer[myid=4]/0.0.0.0:50512:FinalRequestProcessor@415] - shutdown of request processor complete
INFO [CommitProcessor:4:CommitProcessor@150] - CommitProcessor exited loop!
WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:50512:NIOServerCnxn@354] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:50512:NIOServerCnxn@1001] - Closed socket connection for client /... (no session established for client)
INFO [QuorumPeer[myid=4]/0.0.0.0:50512:SyncRequestProcessor@175] - Shutting down
INFO [SyncThread:4:SyncRequestProcessor@155] - SyncRequestProcessor exited!
INFO [QuorumPeer[myid=4]/0.0.0.0:50512:QuorumPeer@670] - LOOKING
{code}

after that all connections to that node initiate, and then are shut down with ZooKeeperServer not running. It seems to stay in this state indefinitely until the process is manually restarted. After that it recovers.

We have seen this happen on multiple servers at the same time resulting in the entire ensemble being unusable.
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
5 years, 10 weeks, 2 days ago 0|i24bev:
ZooKeeper ZOOKEEPER-2105

PrintWriter left unclosed in NIOServerCnxn#checkFourLetterWord

Bug Resolved Minor Not A Problem Unassigned Ted Yu Ted Yu 09/Jan/15 20:11   19/Jan/15 00:22 19/Jan/15 00:22         0 2   {code}
final PrintWriter pwriter = new PrintWriter(
new BufferedWriter(new SendBufferWriter()));
{code}
pwriter should be closed upon return from the method.
9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
5 years, 9 weeks, 3 days ago 0|i247xz:
ZooKeeper ZOOKEEPER-2104

Sudden crash of all nodes in the cluster

Bug Open Major Unresolved Unassigned Benjamin Jaton Benjamin Jaton 09/Jan/15 19:48   09/Dec/16 22:23   3.4.6   server   4 21   In a 3 nodes ensemble, suddenly all the nodes seem to fail, displaying "ZooKeeper is not running" messages.
Not retry seems to be happening after that.

This a request to understand what happened and probably to improve the logs when it does.

See logs below:

NODE1:

-- no log for several days before this --
2015-01-04 16:18:22,259 [myid:1] - WARN [SyncThread:1:FileTxnLog@321] - fsync-ing the write ahead log in SyncThread:1 took 11024ms which will adversely effect operation latency. See the ZooKeeper troubleshooting guide
2015-01-04 16:18:22,380 [myid:1] - WARN [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:Follower@89] - Exception when following the leader
java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:392)
at org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63)
at org.apache.zookeeper.server.quorum.QuorumPacket.deserialize(QuorumPacket.java:83)
at org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:103)
at org.apache.zookeeper.server.quorum.Learner.readPacket(Learner.java:153)
at org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:85)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:786)
2015-01-04 16:18:23,384 [myid:1] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2015-01-04 16:18:23,492 [myid:1] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2015-01-04 16:18:24,060 [myid:1] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running


NODE2:

-- no log for several days before this --
2015-01-04 16:18:21,899 [myid:3] - WARN [QuorumPeer[myid=3]/0:0:0:0:0:0:0:0:2181:Follower@89] - Exception when following the leader
java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:392)
at org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63)
at org.apache.zookeeper.server.quorum.QuorumPacket.deserialize(QuorumPacket.java:83)
at org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:103)
at org.apache.zookeeper.server.quorum.Learner.readPacket(Learner.java:153)
at org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:85)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:786)
2015-01-04 16:18:22,760 [myid:3] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2015-01-04 16:18:22,801 [myid:3] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2015-01-04 16:18:22,886 [myid:3] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running


NODE3 (leader):

-- no log for several days before this --
2015-01-04 16:18:21,897 [myid:2] - WARN [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:LearnerHandler@687] - Closing connection to peer due to transaction timeout.
2015-01-04 16:18:21,898 [myid:2] - WARN [LearnerHandler-/204.53.107.249:43402:LearnerHandler@646] - ******* GOODBYE /204.53.107.249:43402 ********
2015-01-04 16:18:21,905 [myid:2] - WARN [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:LearnerHandler@687] - Closing connection to peer due to transaction timeout.
2015-01-04 16:18:21,907 [myid:2] - WARN [LearnerHandler-/204.53.107.247:45953:LearnerHandler@646] - ******* GOODBYE /204.53.107.247:45953 ********
2015-01-04 16:18:21,918 [myid:2] - WARN [LearnerHandler-/204.53.107.247:45953:LearnerHandler@658] - Ignoring unexpected exception
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1219)
at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:340)
at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:338)
at org.apache.zookeeper.server.quorum.LearnerHandler.shutdown(LearnerHandler.java:656)
at org.apache.zookeeper.server.quorum.LearnerHandler.run(LearnerHandler.java:649)
2015-01-04 16:18:23,003 [myid:2] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2015-01-04 16:18:23,007 [myid:2] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2015-01-04 16:18:23,115 [myid:2] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
9223372036854775807 No Perforce job exists for this issue. 2 9223372036854775807
3 years, 14 weeks, 5 days ago 0|i247xb:
ZooKeeper ZOOKEEPER-2103

ZooKeeper Client Configuration

New Feature Resolved Minor Duplicate Chris Larsen Chris Larsen Chris Larsen 07/Jan/15 15:55   08/Aug/16 10:42 08/Aug/16 10:42 3.6.0 3.5.2 java client   1 3   All I ran into an issue when connecting to two ZooKeeper clusters from the same JVM application. One of the clusters required SASL authentication while the other one did not. Unfortunately the client uses System properties to configure authentication and the client was attempting to authenticate on the non-auth cluster, preventing a connection.

To solve it, I implemented a base config class with helper methods for parsing config settings as well as a client specific subclass that parsed the system system values but allowed for overriding via programatic values or via a file. There are also new Zookeeper constructors to use this config object. I implemented it so that it's completely backwards compatible so it shouldn't break existing installs (and it hasn't yet with my testing).

If folks like this, we could use the same config base for server configs and migrate away from system properties to per object configs. It would also be helpful to centralize more of the "zookeeper.*" strings.

Let me know what ya'll think and thanks!
features, patch 9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
3 years, 32 weeks, 3 days ago 0|i241qv:
ZooKeeper ZOOKEEPER-2102

commitandactivate messages spamming the quorum

Bug Resolved Major Invalid Unassigned Raúl Gutiérrez Segalés Raúl Gutiérrez Segalés 07/Jan/15 14:02   07/Jan/15 18:07 07/Jan/15 18:07     server   0 2   Using zab-dump (https://github.com/twitter/zktraffic/pull/11), I am seeing this in a prod cluster running 3.5.0 + patches:

{noformat}
QuorumPacket(
timestamp=18:45:35:962873,
src=10.0.1.1:2889,
type=commitandactivate,
zxid=292104572694,
length=114
)
QuorumPacket(
timestamp=18:45:35:962876,
src=10.0.1.1:2889,
type=commitandactivate,
zxid=292104572694,
length=114
)
QuorumPacket(
timestamp=18:45:35:962893,
src=10.0.1.1:2889,
type=commitandactivate,
zxid=292104572694,
length=114
)
....
{noformat}

From a ~5min dump, I see ~80k QuorumPackets of which ~50k are commitandactivate packets! Sounds like some sort of loop.

Any ideas [~shralex]?

cc: [~hdeng], [~fpj]
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
5 years, 11 weeks, 1 day ago 0|i241mn:
ZooKeeper ZOOKEEPER-2101

Transaction larger than max buffer of jute makes zookeeper unavailable

Bug Resolved Major Cannot Reproduce Andor Molnar Shaohui Liu Shaohui Liu 04/Jan/15 05:13   19/Dec/19 18:01 26/Mar/18 09:30 3.4.4 3.5.4 jute   0 15   *Problem*
For multi operation, PrepRequestProcessor may produce a large transaction whose size may be larger than the max buffer size of jute. There is check of buffer size in readBuffer method of BinaryInputArchive, but no check in writeBuffer method of BinaryOutputArchive, which will cause that

1, Leader can sync transaction to txn log and send the large transaction to the followers, but the followers failed to read the transaction and can't sync with leader.
{code}
2015-01-04,12:42:26,474 WARN org.apache.zookeeper.server.quorum.Learner: [myid:2] Exception when following the leader
java.io.IOException: Unreasonable length = 2054758
at org.apache.jute.BinaryInputArchive.readBuffer(BinaryInputArchive.java:100)
at org.apache.zookeeper.server.quorum.QuorumPacket.deserialize(QuorumPacket.java:85)
at org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:108)
at org.apache.zookeeper.server.quorum.Learner.readPacket(Learner.java:152)
at org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:85)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:740)
2015-01-04,12:42:26,475 INFO org.apache.zookeeper.server.quorum.Learner: [myid:2] shutdown called
java.lang.Exception: shutdown Follower
at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:166)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:744)
{code}

2, The leader lose all followers, which trigger the leader election. The old leader will become leader again for it has up-to-date data.
{code}
2015-01-04,12:42:28,502 INFO org.apache.zookeeper.server.quorum.Leader: [myid:3] Shutting down
2015-01-04,12:42:28,502 INFO org.apache.zookeeper.server.quorum.Leader: [myid:3] Shutdown called
java.lang.Exception: shutdown Leader! reason: Only 1 followers, need 2
at org.apache.zookeeper.server.quorum.Leader.shutdown(Leader.java:496)
at org.apache.zookeeper.server.quorum.Leader.lead(Leader.java:471)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:753)
{code}
3, The leader can not load the transaction from the txn log for the length of data is larger than the max buffer of jute.
{code}

2015-01-04,12:42:31,282 ERROR org.apache.zookeeper.server.quorum.QuorumPeer: [myid:3] Unable to load database on disk
java.io.IOException: Unreasonable length = 2054758
at org.apache.jute.BinaryInputArchive.readBuffer(BinaryInputArchive.java:100)
at org.apache.zookeeper.server.persistence.Util.readTxnBytes(Util.java:233)
at org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.next(FileTxnLog.java:602)
at org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:157)
at org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:223)
at org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase(QuorumPeer.java:417)
at org.apache.zookeeper.server.quorum.QuorumPeer.getLastLoggedZxid(QuorumPeer.java:546)
at org.apache.zookeeper.server.quorum.FastLeaderElection.getInitLastLoggedZxid(FastLeaderElection.java:690)
at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:737)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:716)
{code}

The zookeeper service will be unavailable until we enlarge the jute.maxbuffer and restart zookeeper hbase cluster.

*Solution*
Add buffer size check in BinaryOutputArchive to avoid large transaction be written to log and sent to followers.

But I am not sure if there are side-effects of throwing an IOException in BinaryOutputArchive and RequestProcessors
9223372036854775807 No Perforce job exists for this issue. 9 9223372036854775807
1 year, 5 weeks, 1 day ago 0|i23xkn:
ZooKeeper ZOOKEEPER-2100

ZooKeeperSaslClient doesn't shut down Login thread

Bug Open Major Unresolved Unassigned Gregory Chanan Gregory Chanan 29/Dec/14 18:59   11/Jan/15 11:54   3.4.5   java client   0 3   I found this in 3.4.5, but a quick perusal of the code suggests this exists in later versions as well.

Setup: I'm running some ZooKeeper SASL tests with Hadoop's MiniKDC under Solr's test framework, which checks for things like thread leaks. The thread leak checker is complaining about the Login thread, which is created but never shut down. It's started here:
https://github.com/apache/zookeeper/blob/6ebd23b32d2cf606e01906bee4460bf79eb7f3fa/src/java/main/org/apache/zookeeper/client/ZooKeeperSaslClient.java#L227 and you can verify via reading the code that it is never shut down.

This may be intentional, because the Login object is static, so it is probably supposed to stick around for the lifetime of the application. This is not great for a test setup, where the idea is that a cluster and all associated clients are started/stopped for each test suite. You wouldn't want either:
1) a thread stick around doing nothing, or
2) sticking around doing something (because it makes the first suite that happens to run behave differently than subsequents suites).

in addition, this only happens with SASL clients, so we'd want to only turn off the leak checker if we are running under SASL (so we don't miss other leaked threads), which is a bit more complexity than I would like.

I'd be happy with a function I could call to say "I'm really done, close down everything, even in the Login thread" or some automatic way of doing it.
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
5 years, 12 weeks, 3 days ago 0|i23ty7:
ZooKeeper ZOOKEEPER-2099

Using txnlog to sync a learner can corrupt the learner's datatree

Bug Patch Available Major Unresolved Martin Kuchta Santeri (Santtu) Voutilainen Santeri (Santtu) Voutilainen 29/Dec/14 16:35   25/Oct/16 16:47   3.5.0, 3.6.0   server   0 7   When a learner sync's with the leader, it is possible for the Leader to send the learner a DIFF that does NOT contain all the transactions between the learner's zxid and that of the leader's zxid thus resulting in a corruption datatree on the learner.
For this to occur, the leader must have sync'd with a previous leader using a SNAP and the zxid requested by the learner must still exist in the current leader's txnlog files.
This issue was introduced by ZOOKEEPER-1413.

*Scenario*
A sample sequence in which this issue occurs:
# Hosts H1 and H2 disconnect from the current leader H3 (crash, network partition, etc). The last zxid on these hosts is Z1.
# Additional transactions occur on the cluster resulting in the latest zxid being Z2.
# Host H1 recovers and connects to H3 to sync and sends Z1 as part of its FOLLOWERINFO or OBSERVERINFO packet.
# The leader, H3, decides to send a SNAP because a) it does not have the necessary records in the in-mem committed log, AND b) the size of the required txnlog to send it larger than the limit.
# Host H1 successfully sync's with the leader (H3). At this point H1's txnlogs have records up to and including Z1 as well as Z2 and up. It does NOT have records between Z1 and Z2.
# Host H3 fails; a leader election occurs and H1 is chosen as the leader
# Host H2 recovers and connects to H1 to sync and sends Z1 in its FOLLOWERINFO/OBSERVERINFO packet
# The leader, H1, determines it can send a DIFF. It concludes this because although it does not have the necessary records in its in-memory commit log, it does have Z1 in its txnlog and the size of the log is less than the limit. H1 ends up with a different size calculation than H3 because H1 is missing all the records between Z1 and Z2 so it has less log to send.
# H2 receives the DIFF and applies the records to its data tree. Depending on the type of transactions that occurred between Z1 and Z2 it may not hit any errors when applying these records.

H2 now has a corrupted view of the data tree because it is missing all the changes made by the transactions between Z1 and Z2.

*Recovery*
The way to recover from this situation is to delete the data/snap directory contents from the affected hosts and have them resync with the leader at which point they will receive a SNAP since they will appear as empty hosts.

*Workaround*
A quick workaround for anyone concerned about this issue is to disable sync from the txnlog by changing the database size limit to 0. This is a code change as it is not a configurable setting.

*Potential fixes*
There are several ways of fixing this. A few of options:
* Delete all snaps and txnlog files on a host when it receives a SNAP from the leader
* Invalidate sync from txnlog after receiving a SNAP. This state must also be persisted on-disk so that the txnlogs with the gap cannot be used to provide a DIFF even after restart. A couple ways in which the state could be persisted:
** Write a file (for example: loggap.<zxid>) in the data dir indicating that the host was sync'd with a SNAP and thus txnlogs might be missing. Presence of these files would be checked when reading txnlogs.
** Write a new record into the txnlog file as "sync'd-by-snap-from-leader" marker. Readers of the txnlog would then check for presence of this record when iterating through it and act appropriately.
9223372036854775807 No Perforce job exists for this issue. 3 9223372036854775807
3 years, 21 weeks, 2 days ago 0|i23tuf:
ZooKeeper ZOOKEEPER-2098

QuorumCnxManager: use BufferedOutputStream for initial msg

Improvement Resolved Major Fixed Raúl Gutiérrez Segalés Raúl Gutiérrez Segalés Raúl Gutiérrez Segalés 29/Dec/14 15:34   03/Jun/15 12:46 29/May/15 17:46 3.5.0 3.5.1, 3.6.0 quorum, server   0 6   Whilst writing fle-dump (a tool like [zk-dump|https://github.com/twitter/zktraffic/], but to dump FastLeaderElection messages), I noticed that QCM is using DataOutputStream (which doesn't buffer) directly.

So all calls to write() are written immediately to the network, which means simple messaages like two participants exchanging Votes can take a couple RTTs! This is specially terrible for global clusters (i.e.: x-country RTTs).

The solution is to use BufferedOutputStream for the initial negotiation between members of the cluster. Note that there are other places were suboptimal (but not entirely unbuffered) writes to the network still exist. I'll get those in separate tickets.

After using BufferedOutputStream we get only 1 RTT for the initial message, so elections & time for for participants to join a cluster is reduced.
9223372036854775807 No Perforce job exists for this issue. 2 9223372036854775807
4 years, 42 weeks, 5 days ago 0|i23ttr:
ZooKeeper ZOOKEEPER-2097

Clarify security requirement for Exists request

Task Patch Available Minor Unresolved Ian Dimayuga Ian Dimayuga Ian Dimayuga 19/Dec/14 17:30   05/Feb/20 07:12   3.4.6, 3.5.0 3.7.0, 3.5.8     0 1   According to the [Programmer's Guide|http://zookeeper.apache.org/doc/current/zookeeperProgrammers.html]:

bq. Everyone implicitly has LOOKUP permission. This allows you to stat a node, but nothing more. (The problem is, if you want to call zoo_exists() on a node that doesn't exist, there is no permission to check.)

This implies that Exists has no security requirement, so the existing comment in FinalRequestProcessor
{code}// TODO we need to figure out the security requirement for this!{code}
can be removed.
9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
1 year, 45 weeks ago 0|i23ndj:
ZooKeeper ZOOKEEPER-2096

C client builds with incorrect error codes in VisualStudio 2010+

Bug Resolved Major Fixed Vitaly Stakhovsky Vitaly Stakhovsky Vitaly Stakhovsky 19/Dec/14 12:41   02/Mar/16 20:30 02/Jun/15 16:44 3.4.6, 3.5.0 3.4.7, 3.5.1, 3.6.0 build, c client   0 5 3600 3600 0% Windows MSVS 2010+ It reports:
warning C4005: 'EWOULDBLOCK' : macro redefinition
warning C4005: 'EINPROGRESS' : macro redefinition

In MSVS 2010+, these constants are now in <errno.h>.
What's worse, they have different numeric values.

Possible fix:
In "src/c/include/winconfig.h" :
#if _MSC_VER < 1600
#define EWOULDBLOCK WSAEWOULDBLOCK
#define EINPROGRESS WSAEINPROGRESS
#endif
0% 0% 3600 3600 9223372036854775807 No Perforce job exists for this issue. 6 9223372036854775807
4 years, 42 weeks, 1 day ago 0|i23mrz:
ZooKeeper ZOOKEEPER-2095

Add Systemd startup/conf files

Improvement Resolved Minor Won't Fix Guillaume ALAUX Guillaume ALAUX Guillaume ALAUX 18/Dec/14 16:58   03/Mar/16 19:32 03/Mar/16 19:32     contrib   1 3   As adoption of systemd by distributions grows, it would be nice to have systemd configuration and startup files for Zookeeper in the upstream tree. I would thus like to contribute the following patch which brings the followings systemd files:

- {{sysusers.d_zookeeper.conf}}: creates {{zookeeper}} Linux system user to run Zookeeper
- {{tmpfiles.d_zookeeper.conf}}: creates temporary {{/var/log/zookeeper}} and {{/var/lib/zookeeper} directories
- {{zookeeper.service}}: regular systemd startup _script_
- {{zookeeper@.service}}: systemd startup _script_ for specific use (for instance when Zookeeper is invoked to support some other piece of software – [example for Kafka|http://pkgbuild.com/git/aur-mirror.git/tree/kafka/systemd_kafka.service#n3], [example for Storm|http://pkgbuild.com/git/aur-mirror.git/tree/storm/systemd_storm-nimbus.service#n3])
9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
4 years, 3 weeks ago 0|i23lfj:
ZooKeeper ZOOKEEPER-2094

ZOOKEEPER-2063 SSL feature on Netty

Sub-task Closed Major Duplicate Ian Dimayuga Ian Dimayuga Ian Dimayuga 05/Dec/14 19:59   19/Dec/19 18:02 13/May/15 13:08 3.4.6, 3.5.0 3.5.2 server   0 7   Add SSL handler to Netty pipeline, and a default X509AuthenticationProvider to perform authentication.

Review board:
https://reviews.apache.org/r/30753/diff/#
9223372036854775807 No Perforce job exists for this issue. 17 9223372036854775807
4 years, 45 weeks, 1 day ago 0|i2354f:
ZooKeeper ZOOKEEPER-2093

Expose cumulative latency and request count

Improvement Patch Available Minor Unresolved Brian Brazil Brian Brazil Brian Brazil 01/Dec/14 13:05   14/Dec/19 06:06     3.7.0 jmx, server   2 4   Currently Zookeeper exposes min, max and average since server start request latency.

It'd also be useful to be able to calculate latency over a given time period. This patch exposes the total latency, and count of number of request. By tracking how these values increase over time, your monitoring system can calculate the average latency over a given period of time.

This only provides milliseconds, as that's what's currently there.


I'm seeing a test failure on ClientPortBindTest.testBindByAddress, this appears to be due to my IPv6 setup.
patch 9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
4 years, 3 weeks ago Expose total request count and latency
Incompatible change
0|i22xlb:
ZooKeeper ZOOKEEPER-2092

A zk instance can not be connected for ZooKeeperServer is not running

Bug Open Major Unresolved Unassigned Shaohui Liu Shaohui Liu 27/Nov/14 08:03   28/Nov/14 06:29   3.4.4       0 3   In our 5 node zk cluster, we found a zk node always can not be connected. From the stack we found the ZooKeeperServer hung at waiting the server to be running. But the node is running normally and synced with the leader.

{code}
$ ./zkCli.sh -server 10.101.10.67:11000 ls /
2014-11-27 20:57:11,843 [myid:] - WARN [main-SendThread(lg-com-master02.bj:11000):ClientCnxn$SendThread@1089] - Session 0x0 for server lg-com-master02.bj/10.101.10.67:11000, unexpected error, closing socket connection and attempting reconnect
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:192)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)
at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:68)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:353)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
Exception in thread "main" org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /
at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1469)
at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1497)
at org.apache.zookeeper.ZooKeeperMain.processZKCmd(ZooKeeperMain.java:726)
at org.apache.zookeeper.ZooKeeperMain.processCmd(ZooKeeperMain.java:594)
at org.apache.zookeeper.ZooKeeperMain.run(ZooKeeperMain.java:355)
at org.apache.zookeeper.ZooKeeperMain.main(ZooKeeperMain.java:283)
{code}

ZooKeeperServer stack
{code}
"NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11000" daemon prio=10 tid=0x00007f60143f7800 nid=0x31fd in Object.wait() [0x00007f5fd4678000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at org.apache.zookeeper.server.ZooKeeperServer.submitRequest(ZooKeeperServer.java:634)
- locked <0x00000007602756a0> (a org.apache.zookeeper.server.quorum.FollowerZooKeeperServer)
at org.apache.zookeeper.server.ZooKeeperServer.submitRequest(ZooKeeperServer.java:626)
at org.apache.zookeeper.server.ZooKeeperServer.createSession(ZooKeeperServer.java:525)
at org.apache.zookeeper.server.ZooKeeperServer.processConnectRequest(ZooKeeperServer.java:841)
at org.apache.zookeeper.server.NIOServerCnxn.readConnectRequest(NIOServerCnxn.java:410)
at org.apache.zookeeper.server.NIOServerCnxn.readPayload(NIOServerCnxn.java:200)
at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:236)
at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
at java.lang.Thread.run(Thread.java:662)
{code}

Any suggestions about this problem? Thanks.
9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
5 years, 16 weeks, 6 days ago 0|i22ulj:
ZooKeeper ZOOKEEPER-2091

Possible logic error in ClientCnxnSocketNIO

Bug Patch Available Major Unresolved Rakesh Radhakrishnan Cheng Cheng 25/Nov/14 20:06   05/Feb/20 07:12   3.4.6 3.7.0, 3.5.8 java client   1 5   When SASL authentication is enabled, the ZooKeeper client will finally call ClientCnxnSocketNIO#sendPacket(Packet p) to send a packet to server:
@Override
void sendPacket(Packet p) throws IOException {
SocketChannel sock = (SocketChannel) sockKey.channel();
if (sock == null) {
throw new IOException("Socket is null!");
}
p.createBB();
ByteBuffer pbb = p.bb;
sock.write(pbb);
}

One problem I can see is that the sock is non-blocking, so when the sock's output buffer is full(theoretically), only part of the Packet is sent out and the communication will break.
9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
3 years, 39 weeks, 2 days ago 0|i22si7:
ZooKeeper ZOOKEEPER-2090

Fix Zookeeper docs "To be done" notices

Bug Open Trivial Unresolved Unassigned Hector Hector 25/Nov/14 09:54   13/Sep/16 14:32       documentation   1 2 86400 86400 0% Official website The docs on the website are full of TBDs since a long time ago. While there is not a full lack of docs, and you can get going with what there is, the main general-purpose entry points are not too polished and they give the impression ZK is not very well maintained to newcomers and anyone who just wants to see how ZK is progressing and refresh concepts.

The ZK overview doc (http://zookeeper.apache.org/doc/trunk/zookeeperOver.html) is supposed to be a first entry point for new Zookeeper users and it is full of _\[tbd\]s_:

{quote}
When the session ends the znode is deleted. Ephemeral nodes are useful when you want to implement \[tbd\].
{quote}
{quote}
And if the connection between the client and one of the Zoo Keeper servers is broken, the client will receive a local notification. These can be used to \[tbd\].
{quote}
{quote}
Timeliness - The clients view of the system is guaranteed to be up-to-date within a certain time bound.

For more information on these, and how they can be used, see \[tbd\]
{quote}
{quote}
For a more in-depth discussion on these, and how they can be used to implement higher level operations, please refer to \[tbd\]
{quote}

{quote}
Some distributed applications have used it to: \[tbd: add uses from white paper and video presentation.\] For more information, see \[tbd\]
{quote}

{quote}
These znodes exists as long as the session that created the znode is active. When the session ends the znode is deleted. Ephemeral nodes are useful when you want to implement \[tbd\].
{quote}

The second entry point, "Getting Started" (http://zookeeper.apache.org/doc/trunk/zookeeperStarted.html)
{quote}
\[tbd: what is the other config param?\]
{quote}

Programmers guide (http://zookeeper.apache.org/doc/trunk/zookeeperProgrammers.html)
{quote}
"If the version it supplies doesn't match the actual version of the data, the update will fail. (This behavior can be overridden. For more information see... )\[tbd...\]"
{quote}
{quote}
Connecting to ZooKeeper

Read Operations

Write Operations

Handling Watches

Miscelleaneous ZooKeeper Operations

Program Structure, with Simple Example

\[tbd\]
{quote}

{quote}
ZooKeeper Whitepaper \[tbd: find url\]

The definitive discussion of ZooKeeper design and performance, by Yahoo! Research
API Reference \[tbd: find url\]

The complete reference to the ZooKeeper API
{quote}


Administration guide (http://zookeeper.apache.org/doc/trunk/zookeeperAdmin.html)
{quote}
Provisioning

Things to Consider: ZooKeeper Strengths and Limitations

Administering

{quote}
{quote}
TBD - tuning options for netty - currently there are none that are netty specific but we should add some. Esp around max bound on the number of reader worker threads netty creates.

TBD - how to manage encryption

TBD - how to manage certificates
{quote}

Since it is not a big deal to fix these, I think it is worth it to spend some hours doing it.
0% 0% 86400 86400 documentation 9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
5 years, 17 weeks, 2 days ago 0|i22rlr:
ZooKeeper ZOOKEEPER-2089

Improve the building blocks part of the programmer's guide

Improvement Open Major Unresolved Flavio Paiva Junqueira Flavio Paiva Junqueira Flavio Paiva Junqueira 25/Nov/14 09:46   05/Feb/20 07:16   3.4.6, 3.5.0 3.7.0, 3.5.8 documentation   0 1   This part of the documentation has been incomplete since 3.1.2. The main reason seems to be that the information that was supposed to be there exists elsewhere in a different form. It needs a revision nonetheless. 9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
5 years, 17 weeks, 2 days ago 0|i22rlb:
ZooKeeper ZOOKEEPER-2088

Provide custom logging hook

Wish Open Trivial Unresolved Jean-François SMIGIELSKI Jean-François SMIGIELSKI Jean-François SMIGIELSKI 24/Nov/14 08:49   05/Feb/20 07:16   3.4.6 3.7.0, 3.5.8 c client   0 1   Linux Hello, proud Zookeeper maintainers!

I incoporated the zookeeper C client API in a quite large code base. The result is fine, everything works as expected but the logs. Our code already manages its log traces via one API, the GLib-2.0 logging features. The current "FILE*" based logging of the C client is not suitable for us.

I propose to integrate a minimal change that would allows me to plug ZK's output on any other : a simple logging hook. I pasted below the patch I propose.

What do you think about ?

diff -r -U3 zookeeper-3.4.6/src/c/include/zookeeper_log.h zookeeper-3.4.6-new/src/c/include/zookeeper_log.h
--- zookeeper-3.4.6/src/c/include/zookeeper_log.h 2014-02-20 11:14:08.000000000 +0100
+++ zookeeper-3.4.6-new/src/c/include/zookeeper_log.h 2014-11-24 13:36:21.088124921 +0100
@@ -44,6 +44,10 @@

FILE* getLogStream();

+typedef void (zk_hook_log) (ZooLogLevel, int, const char*, const char *);
+
+void zoo_set_log_hook (zk_hook_log *hook);
+
#ifdef __cplusplus
}
#endif

diff -r -U3 zookeeper-3.4.6/src/c/src/zk_log.c zookeeper-3.4.6-new/src/c/src/zk_log.c
--- zookeeper-3.4.6/src/c/src/zk_log.c 2014-02-20 11:14:09.000000000 +0100
+++ zookeeper-3.4.6-new/src/c/src/zk_log.c 2014-11-24 14:28:46.151503385 +0100
@@ -122,9 +122,17 @@
return now_str;
}

+static zk_hook_log *log_hook = NULL;
+void zoo_set_log_hook (zk_hook_log *hook)
+{
+ log_hook = hook;
+}
+
void log_message(ZooLogLevel curLevel,int line,const char* funcName,
const char* message)
{
+ if (log_hook) return (*log_hook)(curLevel, line, funcName, message);
+
static const char* dbgLevelStr[]={"ZOO_INVALID","ZOO_ERROR","ZOO_WARN",
"ZOO_INFO","ZOO_DEBUG"};
static pid_t pid=0;



newbie 9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
5 years, 17 weeks, 3 days ago 0|i22prb:
ZooKeeper ZOOKEEPER-2087

Few UX improvements in ZooInspector

Improvement Closed Minor Fixed Adam Dudczak Adam Dudczak Adam Dudczak 21/Nov/14 18:28   21/Jul/16 16:18 09/Mar/16 23:29 3.4.6 3.5.2, 3.6.0 contrib   0 2   A few simple changes would simplify using ZooInspector a lot.

- Alphabetical order of nodes on a tree view
- Short term caching of zookeeper nodes for faster rendering of node tree
- Add/Delete node in context menu of a node
- Keyboard shortcuts for add/deleting a node
- Logging information that ZooInspector failed to load nodeViewers

9223372036854775807 No Perforce job exists for this issue. 2 9223372036854775807
4 years, 2 weeks ago
Reviewed
0|i22ob3:
ZooKeeper ZOOKEEPER-2086

Unnecessary error log when defaultWatcher is not set for ZooKeeper client

Bug Patch Available Minor Unresolved Saurabh Chhajed Cheng Cheng 21/Nov/14 03:37   16/Dec/14 15:50   3.4.6   java client   1 4   In org.apache.zookeeper.ZooKeeper.ZKWatchManager#materialize(), even if the defaultWatcher is null, it is still be added into the Set and returned. This would cause a lot of annoying error log at org.apache.zookeeper.ClientCnxn.EventThread#processEvent as below:
2014-11-21 15:21:23,279 - ERROR - [main-EventThread:ClientCnxn$EventThread@524] - Error while calling watcher
java.lang.NullPointerException
at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)

It can be simply fixed by having a null check in ZKWatchManager.
9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
5 years, 14 weeks, 2 days ago 0|i22n5b:
ZooKeeper ZOOKEEPER-2085

Upgrade to jline2

Improvement Resolved Major Duplicate Unassigned Brock Noland Brock Noland 18/Nov/14 19:54   18/Nov/14 19:56 18/Nov/14 19:56         0 1   Hive has upgraded to jline2 in HIVE-8609 due to a serious bug we found in HIVE-8565. It'd be great if Zookeeper could upgrade as well. 9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
5 years, 18 weeks, 1 day ago 0|i22j93:
ZooKeeper ZOOKEEPER-2084

Document local session parameters

Improvement Closed Major Resolved Unassigned Flavio Paiva Junqueira Flavio Paiva Junqueira 15/Nov/14 11:04   14/Feb/20 10:23 10/Oct/19 23:37 3.5.0 3.6.0, 3.5.7 documentation   1 3   Document the options introduced in ZOOKEEPER-1147. 9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
22 weeks, 6 days ago 0|i22fb3:
ZooKeeper ZOOKEEPER-2083

Remove deprecated LE implementations

Improvement Resolved Major Fixed Enrico Olivelli Flavio Paiva Junqueira Flavio Paiva Junqueira 15/Nov/14 05:38   09/Jan/20 05:29 09/Jan/20 05:27   3.6.0     0 2 0 12600   As per ZOOKEEPER-1153, we should remove implementations 0, 1, 2. 100% 100% 12600 0 pull-request-available 9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
10 weeks ago 0|i22f73:
ZooKeeper ZOOKEEPER-2082

Mistype of electionAlgo can fill out your disk in minutes

Bug Resolved Minor Fixed Unassigned Tianyin Xu Tianyin Xu 14/Nov/14 17:42   21/Oct/18 03:28 21/Oct/18 03:28 3.4.6   leaderElection   0 6   Cluster (multi-server) setup The parameter, electionAlgo, is supposed to be 0--3. However, when I mistyped the value in my zoo.cfg (I'm stupid), ZK falls into a dead loop and starts filling up the entire disk which millions of the follow 2 lines...

2014-11-14 14:28:44,588 \[myid:3\] - INFO \[QuorumPeer\[myid=3\]/0:0:0:0:0:0:0:0:2183:QuorumPeer@714\] - LOOKING
2014-11-14 14:28:44,588 \[myid:3\] - WARN \[QuorumPeer\[myid=3\]/0:0:0:0:0:0:0:0:2183:QuorumPeer@764\] - Unexpected exception
java.lang.NullPointerException
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:762)

The error rooted in createElectionAlgorithm() where an invalid setting leads to null for the Election object. Then, in the while look in run(), it causes null-pointer de-referencing which is captured but is not handled well.

I think our should check the setting of electionAlg in the very beginning to make sure it's a valid setting, instead of using it at runtime and cause the unfortunate things.

Let me know if you wanna a patch. I'd like to check it in the parseProperties() function in QuorumPeerConfig.java.

Thanks!
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
2 years, 29 weeks, 6 days ago 0|i22enz:
ZooKeeper ZOOKEEPER-2081

Leader election cannot complete when a node is blackholed (unreachable) even when quorum is possible.

Bug Open Major Unresolved Unassigned Matan Matan 13/Nov/14 20:27   29/Oct/15 14:33   3.3.6, 3.4.6   leaderElection, quorum   0 13   Verified on RHEL and Mac OS X. I noticed a situation when one of our 3-node clusters on RHEL lost a machine due to PSU failure. The remaining two nodes failed to complete leader election and would continually restart the leader election process.
Restarting the nodes would not help and they would reach the same exact state.

This was curious so I spent some time and managed to reproduce this on my local machine and found what looks like the main factor:
When a node is unreachable (timeouts), this somehow causes the election process to get out of sync. Once a leader is decided, the follower tries to connect to the leader only when the leader is not listening.
Then the follower gives up and the process starts again ad infinitum.

How to reproduce on a local machine:

1. Setup up a 3 node cluster of ZK. Note we only need to set up 2 boxes since we'll just make the third unreachable:

MyId 1:

server.1=MyMachine:2881:3881
server.2=<Put any IP that we can block>:2882:3882
server.3=MyMachine:2883:3883

MyId 3:

server.1=MyMachine:2881:3881
server.2=<Put any IP that we can block>:2882:3882
server.3=MyMachine:2883:3883

Now set up a blackhole route for the IP you choose (Mac OSX, Linux is similar):
> route add -host <IP you selected> 127.0.0.1 -blackhole

Start your 2 nodes. They will never reach quorum.

However, if I remove the blackhole route and just not start the 3rd instance (but the host is still reachable), it will work fine and quorum will be reached almost immediately.

It seems the difference between the “timeout” and a "connection refused” makes all the difference somehow in the election process.

I verified this behavior on 3.4.6 and 3.3.6.
9223372036854775807 No Perforce job exists for this issue. 2 9223372036854775807
4 years, 21 weeks ago 0|i22cz3:
ZooKeeper ZOOKEEPER-2080

ZOOKEEPER-2135 Fix deadlock in dynamic reconfiguration

Sub-task Closed Major Fixed Michael Han Ted Yu Ted Yu 12/Nov/14 18:54   17/May/17 23:43 09/Feb/17 00:10 3.5.2 3.5.3, 3.6.0 server   0 11   I got the following test failure on MacBook with trunk code:
{code}
Testcase: testCurrentObserverIsParticipantInNewConfig took 93.628 sec
FAILED
waiting for server 2 being up
junit.framework.AssertionFailedError: waiting for server 2 being up
at org.apache.zookeeper.server.quorum.ReconfigRecoveryTest.testCurrentObserverIsParticipantInNewConfig(ReconfigRecoveryTest.java:529)
at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:52)
{code}
9223372036854775807 No Perforce job exists for this issue. 9 9223372036854775807
2 years, 45 weeks, 5 days ago 0|i22b33:
ZooKeeper ZOOKEEPER-2079

Stop daemon with "kill" rather than "kill -9"

Improvement Resolved Minor Fixed Guillaume ALAUX Guillaume ALAUX Guillaume ALAUX 12/Nov/14 12:08   12/Oct/17 04:45 17/Nov/14 01:50   3.5.1, 3.6.0 scripts   0 4   *nix Script `zkServer.sh` stops zookeeper by sending the java process a `kill -9` (SIGKILL). As there seems to be no technical reasons to use such a radical signal rather than the default SIGTERM (-15), I would propose to just use `kill` rather than `kill -9`.

My use case is for Systemd service files for Zookeeper which always consider Zookeeper java process as failing when a clean `stop` is issued.

Systemd output showing this "fail":
---------------8<---------------
# sudo systemctl status zookeeper.service
● zookeeper.service - Highly reliable distributed coordination server
Loaded: loaded (/usr/lib/systemd/system/zookeeper.service; disabled)
Active: failed (Result: signal) since Wed 2014-11-05 11:23:29 CET; 2s ago
Process: 656 ExecStop=/usr/bin/zkServer.sh stop (code=exited,
status=0/SUCCESS)
Process: 406 ExecStart=/usr/bin/zkServer.sh start (code=exited,
status=0/SUCCESS)
Main PID: 414 (code=killed, signal=KILL)

Nov 05 11:23:29 magenta zookeeper[656]: Stopping zookeeper ... STOPPED
Nov 05 11:23:29 magenta systemd[1]: zookeeper.service: main process
exited, code=killed, status=9/KILL
Nov 05 11:23:29 magenta systemd[1]: Unit zookeeper.service entered failed state.
---------------8<---------------

There is no way to make this `status=9/KILL` to be recognized by
Systemd as a regular exit code, even with `SuccessExitStatus=9 KILL
SIGKILL`. On the other hand, turning this `kill -9` into a regular
`kill` (-15 implied) makes it:

---------------8<---------------
# sudo systemctl status zookeeper.service
● zookeeper.service - Highly reliable distributed coordination server
Loaded: loaded (/usr/lib/systemd/system/zookeeper.service; disabled)
Active: inactive (dead)

Nov 05 11:14:27 magenta zookeeper[30032]: Using config:
/usr/share/zookeeper/bin/../conf/zoo.cfg
Nov 05 11:14:27 magenta zookeeper[30032]: Stopping zookeeper ... STOPPED
---------------8<---------------
9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
2 years, 23 weeks ago Kill java process with `SIGTERM` rather than `SIGKILL` 0|i22afz:
ZooKeeper ZOOKEEPER-2078

zkServer.sh uses pattern unsupported by "grep" on Solaris

Bug Resolved Minor Duplicate Chris Nauroth metatech metatech 12/Nov/14 07:09   01/May/15 01:57 01/May/15 01:57 3.4.5   scripts   0 2   Solaris 11 The script "zkServer.sh" contains a pattern (POSIX "character class syntax") which is not supported by "grep" on Solaris (both versions 10 and 11).

{code}
ZOO_DATADIR="$(grep "^[[:space:]]*dataDir" "$ZOOCFG" | sed -e 's/.*=//')"
{code}

This results into the environment variable being set with an empty value, which later gives the following error :

{code}
Starting zookeeper ... bin/zkServer.sh: line 114: /zookeeper_server.pid: Permission denied
{code}


The workaround is to simplify the pattern used by "grep" :

{code}
ZOO_DATADIR="$(grep "^dataDir" "$ZOOCFG" | sed -e 's/.*=//')"
{code}

The same pattern is also used in the "status" command, which fails to read the "clientPort", which results into the following error :
{code}
Error contacting service. It is probably not running.
{code}
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
4 years, 46 weeks, 6 days ago 0|i22a1r:
ZooKeeper ZOOKEEPER-2077

Wild-card/Regex Support for Zookeeper client commands

New Feature Open Major Unresolved Unassigned Vivek Madani Vivek Madani 12/Nov/14 03:44   17/Nov/15 01:14       java client   2 5   We had an use-case where we had to list nodes matching a particular pattern from a given path. While looking at the ZK client commands, it seems that it does not support wildcard/regex.

I did try to overcome this by making some basic changes to the LSCommand.java and adding a "-m" switch which accepts regex. Since I implemented this using java.util.regex, it supports everything that Java regex supports.

I was thinking such functionality can be useful for 'ls' as well as 'delete' (and deleteall). Though I implemented this at the client code for ls - this can be done at the server side code as well and I have a preliminary plan on top of my head to do this for ls, delete, deleteall.

Will it be worthwhile addition to make to zookeeper client? If so, I can work on submitting a patch.

Points to consider in case such a support can be implemented:
1. Do we support Java regex or Unix Shell wildcards ( * )?
2. Right now, create allows creating nodes with characters like * - we need to make sure that such a change does not break or create confusion (Unix too allows creating a directory with * BTW).

Any thoughts on whether this will be a worthwhile addition to Zookeeper client?
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
4 years, 18 weeks, 2 days ago 0|i229u7:
ZooKeeper ZOOKEEPER-2076

Improve Leader Change Mechanism

Improvement Open Major Unresolved Atri Sharma Alexander Shraer Alexander Shraer 10/Nov/14 18:59   31/Mar/17 13:46   3.5.0   server   0 5   When a leader is removed during a reconfiguration, ZOOKEEPER-107 uses a mechanism where the old leader nominates the new one. Although it reduces the time for a new leader to be elected, it still takes too long. This JIRA is for two things:
1. Improve the mechanism, e.g., avoid loading snapshots, etc. during the handoff.
2. Make it a first-class citizen & export it as a client API. We get questions about this once in a while - how do I cause a different leader to be elected ? Currently the response is either kill or reconfigure the current leader.

Any one interested to work on this ?
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
2 years, 50 weeks, 6 days ago 0|i227if:
ZooKeeper ZOOKEEPER-2075

Utils.bufEquals

Bug Open Major Unresolved Unassigned Charlie Helin Charlie Helin 07/Nov/14 15:22   07/Nov/14 15:22   3.4.6   java client   0 1   Just happened to notice Utils.bufEquals(byte[], byte[]) as a rather large outlier (~7% CPU time) when running with an attached profiler.

By just simply switching the implementation to delegate directly to Arrays.equals(byte[], byte[]) the invocation disappears from the profile. The reason for this is that this is one of the methods which the JIT (not the interpreter) will generate an intrinsic for, using the builtin support of the CPU to do the check.

The fix is trivial

{code}
public static boolean bufEquals(byte onearray[], byte twoarray[] ) {
return Arrays.equals(onearray, twoarray);
}
{code}
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
5 years, 19 weeks, 6 days ago 0|i224gf:
ZooKeeper ZOOKEEPER-2074

Incorrect exit codes for "./zkCli.sh cmd arg"

Bug Closed Minor Fixed Abraham Fine Surendra Singh Lilhore Surendra Singh Lilhore 07/Nov/14 04:20   17/May/17 23:44 10/Aug/16 13:33 3.5.0 3.5.3, 3.6.0     0 10   Linux@hghoulaslx406:/> $ZOOKEEPER_HOME/bin/zkCli.sh create /test "test"
Created /test1
Linux@hghoulaslx406:/> echo $?
0
Linux@hghoulaslx406:/> $ZOOKEEPER_HOME/bin/zkCli.sh create /test "test"
Node already exists: /test1
Linux@hghoulaslx406:/> echo $?
0
Linux@hghoulaslx406:/> $ZOOKEEPER_HOME/bin/zkCli.sh delete /test
Linux@hghoulaslx406:/> echo $?
0
Linux@hghoulaslx406:/> $ZOOKEEPER_HOME/bin/zkCli.sh delete /test
Node does not exist: /test1
Linux@hghoulaslx406:/> echo $?
0


Here for failed command it should return exit code 1
9223372036854775807 No Perforce job exists for this issue. 5 9223372036854775807
3 years, 32 weeks, 1 day ago
Reviewed
0|i223lz:
ZooKeeper ZOOKEEPER-2073

Memory leak on zookeeper_close

Bug Resolved Critical Fixed Dave Gosselin Dave Gosselin Dave Gosselin 04/Nov/14 12:10   22/Feb/15 17:52 22/Feb/15 17:09 3.6.0 3.4.7, 3.5.1, 3.6.0 c client   0 5   When running valgrind against a zookeeper client using the C API, I noticed an occasional memory leak on zookeeper_close. I traced the issue to a regression added by fix ZOOKEEPER-804. The attached patch fixes the regression and the associated memory leak. 9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
5 years, 4 weeks, 4 days ago 0|i21y9j:
ZooKeeper ZOOKEEPER-2072

Netty Server Should Configure Child Channel Pipeline By Specifying ChannelPipelineFactory

Bug Resolved Major Fixed Hongchao Deng Hongchao Deng Hongchao Deng 28/Oct/14 18:19   30/Jan/15 06:13 29/Jan/15 21:26   3.5.1, 3.6.0 server   0 6   Currently, netty server is setting up child channel in this way:
{code}
bootstrap.getPipeline().addLast("servercnxnfactory", channelHandler);
{code}

According to the [netty doc|http://netty.io/3.9/api/org/jboss/netty/bootstrap/ServerBootstrap.html],
bq. you cannot use this approach if you are going to open more than one Channels or run a server that accepts incoming connections to create its child channels.
9223372036854775807 No Perforce job exists for this issue. 3 9223372036854775807
5 years, 7 weeks, 6 days ago 0|i21p3b:
ZooKeeper ZOOKEEPER-2071

Update docs for Apache branding and trademark requirements

Bug Open Major Unresolved Wendy Smoak Wendy Smoak Wendy Smoak 26/Oct/14 19:38   08/Nov/14 06:18   3.5.0       0 2   The documentation needs to follow the Apache branding and trademark requirements: http://www.apache.org/foundation/marks/pmcs.html

9223372036854775807 No Perforce job exists for this issue. 3 9223372036854775807
5 years, 19 weeks, 5 days ago
Incompatible change
0|i21le7:
ZooKeeper ZOOKEEPER-2070

Explain how to update and publish the project website

Task Open Minor Unresolved Unassigned Wendy Smoak Wendy Smoak 26/Oct/14 19:30   26/Oct/14 19:34           0 1   Need to document how the different parts of the project website get updated and published, both for committers, and for potential contributors who do not have write access. 9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
5 years, 21 weeks, 4 days ago 0|i21ldz:
ZooKeeper ZOOKEEPER-2069

ZOOKEEPER-2063 Netty Support for ClientCnxnSocket

Sub-task Resolved Major Fixed Hongchao Deng Hongchao Deng Hongchao Deng 26/Oct/14 12:58   29/Nov/16 00:49 20/Dec/14 09:37   3.5.1, 3.6.0     0 7   Review Board: https://reviews.apache.org/r/27244/diff/# 9223372036854775807 No Perforce job exists for this issue. 21 9223372036854775807
5 years, 13 weeks, 4 days ago 0|i21l7j:
ZooKeeper ZOOKEEPER-2068

ServerCnxnTest.testServerCnxnExpiry failed when using Netty server option

Bug Open Major Unresolved Unassigned Hongchao Deng Hongchao Deng 25/Oct/14 23:38   01/Nov/14 11:57           0 1   If using Netty server (setting "zookeeper.serverCnxnFactory" to "NettyServerCnxnFactory"), ServerCnxnTest.testServerCnxnExpiry always failed 9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
5 years, 21 weeks, 4 days ago 0|i21kzb:
ZooKeeper ZOOKEEPER-2067

Init script fails to track PID file when using a non-standard dataDir

Bug Patch Available Minor Unresolved Jeremy Carroll Jeremy Carroll Jeremy Carroll 24/Oct/14 21:40   03/Mar/16 01:14   3.4.6, 3.5.0       1 2   When setting a dataDir in zoo.cfg that does not match /var/lib/zookeeper, the supplied init.d script failed to track the PID file. This change moves the logic that is present in zkServer.sh to determine the PID location into zkEnv.sh. Also removed the hard coded path for the zookeeper dataDir. debian 9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
4 years, 3 weeks ago packaging 0|i21kkv:
ZooKeeper ZOOKEEPER-2066

Updates to README.txt

Improvement Resolved Minor Fixed Camille Fournier Wendy Smoak Wendy Smoak 24/Oct/14 11:16   25/Oct/14 07:25 24/Oct/14 22:39   3.5.1     0 3   Updates to README.txt
- first reference should be to Apache ZooKeeper
- fix obsolete ibiblio-rsync-repository url
- better describe the release process
- minor grammar and punctuation changes

9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
5 years, 21 weeks, 5 days ago 0|i21jqf:
ZooKeeper ZOOKEEPER-2065

Debian package java depdencency

Improvement Resolved Minor Won't Fix Unassigned Jeremy Carroll Jeremy Carroll 22/Oct/14 20:05   03/Mar/16 11:18 03/Mar/16 11:17 3.4.6, 3.5.0       0 1   The Debian control file sets a dependency on sun-java6-jre. We currently run Zookeeper in production with Java 1.7. This makes it difficult to support in our environment since it attempts to install an older JRE upon package installation.

I propose that we change this line from sun-java6-jre to default-jre. Then the operator of the system can choose which Java version to run with.
debian 9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
4 years, 3 weeks ago packaging 0|i21h4n:
ZooKeeper ZOOKEEPER-2064

Prevent resource leak in various classes

Bug Resolved Critical Fixed Ted Yu Ted Yu Ted Yu 21/Oct/14 15:57   30/Nov/14 06:23 29/Nov/14 10:57   3.4.7, 3.5.1, 3.6.0     0 4   In various classes, there is potential resource leak.
e.g. LogIterator / RandomAccessFileReader is not closed upon return from the method.

Corresponding close() should be called to prevent resource leak.
9223372036854775807 No Perforce job exists for this issue. 3 9223372036854775807
5 years, 16 weeks, 4 days ago 0|i21f1r:
ZooKeeper ZOOKEEPER-2063

Netty+SSL support for client-server communication

New Feature Open Major Unresolved Unassigned Hongchao Deng Hongchao Deng 17/Oct/14 16:34   24/Oct/17 00:46           1 10   ZOOKEEPER-2069, ZOOKEEPER-2094, ZOOKEEPER-2122 ZooKeeper currently have netty option on server side. We want to support netty on client side too. After that, we could add ssl support based on netty channel. 100% 51600 0 9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
5 years, 5 weeks, 1 day ago 0|i21b4f:
ZooKeeper ZOOKEEPER-2062

RemoveWatchesTest takes forever to run

Bug Resolved Major Fixed Chris Nauroth Flavio Paiva Junqueira Flavio Paiva Junqueira 14/Oct/14 19:23   16/Dec/18 09:27 04/May/15 22:44 3.5.0 3.5.1, 3.6.0 tests   0 7   [junit] Running org.apache.zookeeper.RemoveWatchesTest
[junit] Tests run: 46, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 306.188 sec
remove_watches 9223372036854775807 No Perforce job exists for this issue. 4 9223372036854775807
4 years, 46 weeks, 2 days ago 0|i216dr:
ZooKeeper ZOOKEEPER-2061

Zookeeper needs an official RPM package. I am happy to build and submit one.

Wish Resolved Major Won't Fix Unassigned Brian Weber Brian Weber 14/Oct/14 14:51   03/Mar/16 11:24 03/Mar/16 11:24     build   1 3   9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
4 years, 3 weeks ago 0|i2160v:
ZooKeeper ZOOKEEPER-2060

Trace bug in NettyServerCnxnFactory

Bug Resolved Major Fixed Ian Dimayuga Ian Dimayuga Ian Dimayuga 14/Oct/14 13:58   30/Nov/14 06:23 19/Nov/14 17:40 3.4.6, 3.5.0 3.4.7, 3.5.1, 3.6.0 server   0 5 86400 86400 0% In NettyServerCnxnFactory, high throughput triggers a deadlock.

This is caused by a channel-buffer-dumping debug statement in NettyServerCnxnFactory.java that is executed regardless of log level.

This code path only executes when the server is throttling, but when it does it encounters a race and occasional deadlock between the channel buffer and NettyServerCnxn (jstack attached).

The proposed fix adds the debug logging guard to this statement, similar to other existing statements.
0% 0% 86400 86400 9223372036854775807 No Perforce job exists for this issue. 3 9223372036854775807
5 years, 16 weeks, 4 days ago 0|i215xb:
ZooKeeper ZOOKEEPER-2059

Use command like this "./zkCli.sh -server host:port cmd args" but it doesn't work, 3.4.5 version is work fine

Bug Resolved Major Cannot Reproduce Unassigned huanghaijun huanghaijun 09/Oct/14 05:03   26/Aug/15 16:48 26/Aug/15 16:48 3.4.6       0 3   Use command like this [./zkCli.sh -server host:port cmd args], such as [./zkCli.sh -server localhost:2181 create /test ""] to create a node, 3.4.5 is work fine, but 3.4.6 it doesn't work.

for 3.4.5 it is ok
zookeeper-3.4.5/bin> ./zkCli.sh -server localhost:34096 create /test ""
Connecting to localhost:34096

WATCHER::

WatchedEvent state:SyncConnected type:None path:null
Created /test

for 3.4.6 it's not ok
zookeeper-3.4.6/bin> ./zkCli.sh -server localhost:43096 crate /test1 ""
Connecting to localhost:43096
....
2014-10-10 01:24:44,517 [myid:] - INFO [main:ZooKeeper@438] - Initiating client connection, connectString=localhost:43096 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@48b8f82d

9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
4 years, 30 weeks, 1 day ago 0|i20z7j:
ZooKeeper ZOOKEEPER-2058

rat: exclude *.cer files

Bug Resolved Major Fixed Michi Mutsuzaki Michi Mutsuzaki Michi Mutsuzaki 09/Oct/14 01:04   13/Oct/14 00:22 12/Oct/14 23:20   3.6.0 build   1 5   Somehow the release audit started complaining about *.cer files. 9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
5 years, 23 weeks, 3 days ago 0|i20yz3:
ZooKeeper ZOOKEEPER-2057

ZooKeeper should be as fast as Kafka

Wish Open Major Unresolved Unassigned Hongchao Deng Hongchao Deng 08/Oct/14 20:52   08/Oct/14 20:52           0 1   Kafka can achieve ~750K/s records for async replication. We can see that it's highly performant for eventual consistency model.

Some might argue that ZooKeeper is a strong consistency model. Nonetheless there is nuance -- ZooKeeper can read stale data. Let's say that stale data is "consistent data". ZK can do async replication and provide only consistent data.

This can be easily achieved by MVCC database design. There might be other benefits too, e.g. watcher can now have some kind of version and reconnection won't incur data loss; multi can rollback easily to older version.

However, this requires change no less than rewriting ZK. Just raise this topic up and see what people think.
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
5 years, 24 weeks ago 0|i20ypr:
ZooKeeper ZOOKEEPER-2056

Zookeeper 3.4.x and 3.5.0-alpha is not OSGi compliant

Bug Resolved Major Fixed Deiwin Sarjas Keren Dong Keren Dong 08/Oct/14 11:51   21/Aug/15 12:27 06/Apr/15 21:55 3.4.6, 3.5.0 3.4.7, 3.5.1, 3.6.0     2 9   Similar to this issue https://issues.apache.org/jira/browse/ZOOKEEPER-1334, the MANIFEST.MF is still flawed. When using in OSGi, I got this exception:

java.lang.NoClassDefFoundError: org/ietf/jgss/GSSException
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1063)[168:org.apache.hadoop.zookeeper:3.5.01]
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1114)[168:org.apache.hadoop.zookeeper:3.5.01]
Caused by: java.lang.ClassNotFoundException: org.ietf.jgss.GSSException not found by org.apache.hadoop.zookeeper [168]
at org.apache.felix.framework.BundleWiringImpl.findClassOrResourceByDelegation(BundleWiringImpl.java:1532)[org.apache.felix.framework-4.2.1.jar:]
at org.apache.felix.framework.BundleWiringImpl.access$400(BundleWiringImpl.java:75)[org.apache.felix.framework-4.2.1.jar:]
at org.apache.felix.framework.BundleWiringImpl$BundleClassLoader.loadClass(BundleWiringImpl.java:1955)[org.apache.felix.framework-4.2.1.jar:]
at java.lang.ClassLoader.loadClass(ClassLoader.java:356)[:1.7.0_15]
... 2 more

Looking at the bundle headers, it doesn't have the package org.ietf.jgss imported:

Import-Package =
javax.management;resolution:=optional,
javax.security.auth.callback,
javax.security.auth.login,
javax.security.sasl,
org.slf4j;version="[1.6,2)",
org.jboss.netty.buffer;resolution:=optional;version="[3.2,4)",
org.jboss.netty.channel;resolution:=optional;version="[3.2,4)",
org.jboss.netty.channel.group;resolution:=optional;version="[3.2,4)",
org.jboss.netty.channel.socket.nio;resolution:=optional;version="[3.2,4)",
org.osgi.framework;resolution:=optional;version="[1.5,2)",
org.osgi.util.tracker;resolution:=optional;version="[1.4,2)"
Export-Package =
org.apache.zookeeper;version=3.5.01,
org.apache.zookeeper.client;version=3.5.01,
org.apache.zookeeper.data;version=3.5.01,
org.apache.zookeeper.version;version=3.5.01,
org.apache.zookeeper.server;version=3.5.01,
org.apache.zookeeper.server.auth;version=3.5.01,
org.apache.zookeeper.server.persistence;version=3.5.01,
org.apache.zookeeper.server.quorum;version=3.5.01,
org.apache.zookeeper.common;version=3.5.01
easyfix 9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
4 years, 50 weeks, 2 days ago 0|i20xsv:
ZooKeeper ZOOKEEPER-2055

Don't throw ArrayIndexOutOfBoundsException when SASL username/password isn't specified

Bug Patch Available Minor Unresolved Steve R Steve R Steve R 07/Oct/14 07:56   03/Mar/16 01:06           0 1   When using SASLAuthenticationProvider and the jaas.conf file doesn't have a username and/or password for either the server or client configuration, when the client tries to connect via zkCli, an ArrayIndexOutOfBoundsException is thrown

Example conf file:
Server {
org.apache.zookeeper.server.auth.DigestLoginModule required;
};
Client {
org.apache.zookeeper.server.auth.DigestLoginModule required
username="bob"
password="bob123";
};

Shows the resuting information:
INFO [main-SendThread(127.0.0.1:2181)] Client will use DIGEST-MD5 as SASL mechanism.
ERROR[main-SendThread(127.0.0.1:2181)] Exception while trying to create SASL client: java.lang.ArrayIndexOutOfBoundsException: Array index out of range: 0
9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
4 years, 3 weeks ago 0|i20vnz:
ZooKeeper ZOOKEEPER-2054

test-patch.sh: don't set ulimit -n

Bug Resolved Major Fixed Michi Mutsuzaki Michi Mutsuzaki Michi Mutsuzaki 06/Oct/14 21:52   08/Oct/14 06:30 08/Oct/14 04:18   3.6.0     0 4   It seems to be causing NioNettySuiteHammerTest failure. pre-commit 9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
5 years, 24 weeks, 1 day ago 0|i20v9j:
ZooKeeper ZOOKEEPER-2053

Zookeeper scripts should honor ZOOKEEPER_HOME

Bug Patch Available Major Unresolved Unassigned Owen O'Malley Owen O'Malley 02/Oct/14 16:39   02/Oct/14 18:50           0 1   Currently the scripts will determine the root of the Zookeeper installation based on the location of the script. However, it would be convenient if the scripts honored the ZOOKEEPER_HOME environment variable like the other Hadoop-related projects. 9223372036854775807 No Perforce job exists for this issue. 2 9223372036854775807
5 years, 25 weeks ago 0|i20qyv:
ZooKeeper ZOOKEEPER-2052

Unable to delete a node when the node has no children

Bug Resolved Major Fixed Hongchao Deng Yip Ng Yip Ng 02/Oct/14 03:08   28/Oct/14 07:10 28/Oct/14 00:41 3.4.6, 3.5.0 3.4.7, 3.5.1, 3.6.0 server   0 8   Red Hat Enterprise Linux 6.1 x86_64, standalone or 3 node ensemble (v3.4.6), 2 Java clients (v3.4.6) We stumbled upon a ZooKeeper bug where a node with no children cannot be removed on our 3 node ZooKeeper ensemble or standalone ZooKeeper on Red Hat Enterprise Linux x86_64 environment. Here is an example scenario/setup:

o Standalone ZooKeeper or 3 node ensemble (v3.4.6)
o 2 Java clients (v3.4.6)
- Client A creates a persistent node (e.g.: /metadata/resources)
- Client B creates ephemeral nodes under this persistent node

o Client A attempts to remove the /metadata/resources node via multi op
delete but fails since there are children
o Client B's session expired, all the ephemeral nodes are removed
o Client A attempts to recursively remove /metadata/resources node via
multi op, this is expected to succeed but got the following exception:
org.apache.zookeeper.KeeperException$NotEmptyException:
KeeperErrorCode = Directory not empty

(Note that Client B is the only client that creates these ephemeral nodes)

o After this, we use zkCli.sh to inspect the problematic node but the zkCli.sh shows the /metadata/resources node indeed have no children but it will not allow /metadata/resources node to get deleted. (shown below)

[zk: localhost:2181(CONNECTED) 0] ls /
[zookeeper, metadata]
[zk: localhost:2181(CONNECTED) 1] ls /metadata
[resources]
[zk: localhost:2181(CONNECTED) 2] get /metadata/resources
null
cZxid = 0x3
ctime = Wed Oct 01 22:04:11 PDT 2014
mZxid = 0x3
mtime = Wed Oct 01 22:04:11 PDT 2014
pZxid = 0x9
cversion = 2
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 0
numChildren = 0
[zk: localhost:2181(CONNECTED) 3] delete /metadata/resources
Node not empty: /metadata/resources
[zk: localhost:2181(CONNECTED) 4] get /metadata/resources
null
cZxid = 0x3
ctime = Wed Oct 01 22:04:11 PDT 2014
mZxid = 0x3
mtime = Wed Oct 01 22:04:11 PDT 2014
pZxid = 0x9
cversion = 2
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 0
numChildren = 0

o The only ways to remove this node is to either:
a) Restart the ZooKeeper server
b) set data to /metadata/resources then followed by a subsequent delete.
9223372036854775807 No Perforce job exists for this issue. 11 9223372036854775807
5 years, 21 weeks, 2 days ago 0|i20pr3:
ZooKeeper ZOOKEEPER-2051

Creating ephemeral znodes from within a transaction fail with local sessions

Bug Open Major Unresolved Raúl Gutiérrez Segalés Raúl Gutiérrez Segalés Raúl Gutiérrez Segalés 30/Sep/14 16:08   14/Dec/19 06:06   3.5.0 3.7.0 server   0 5   With local sessions enabled, the premise is that as soon as you try to create an ephemeral znode your session will be upgraded to global. The problem is that the session upgrade logic doesn't intercept transactions. So creating an ephemeral znode from within a transaction fails with SessionExpired.

A small example with Kazoo:

{noformat}
from kazoo.client import KazooClient

k = KazooClient("localhost:2181")
k.start()

t = k.transaction()
t.create("/hello_", "", ephemeral=True)
t.commit()
[kazoo.exceptions.SessionExpiredError((), {})]
{noformat}

A workaround, for now, is to create an ephemeral before your transaction which forces your session to be upgraded.

Possible solutions could be:

* extending zookeeper_init() so that you can request global=True
* and/or, providing an upgradeSession() API

Thoughts?

cc: [~thawan], [~phunt], [~fpj]


9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
5 years, 21 weeks, 6 days ago 0|i20ngf:
ZooKeeper ZOOKEEPER-2050

Maven dependency should remove the 3 dependencies

Bug Open Minor Unresolved Unassigned samraj samraj 30/Sep/14 09:35   30/Sep/14 09:35   3.5.0       0 2   Just add the latest zookeeper version in the maven dependency When i add the latest zookeeper jar in the dependency it throws the error and say following jars are missing.If i added those in exclusion its working fine.

<exclusions>
<exclusion>
<groupId>com.sun.jmx</groupId>
<artifactId>jmxri</artifactId>
</exclusion>
<exclusion>
<groupId>com.sun.jdmk</groupId>
<artifactId>jmxtools</artifactId>
</exclusion>
<exclusion>
<groupId>javax.jms</groupId>
<artifactId>jms</artifactId>
</exclusion>
</exclusions>
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
5 years, 25 weeks, 2 days ago 0|i20mun:
ZooKeeper ZOOKEEPER-2049

Yosemite build failure: htonll conflict

Bug Resolved Major Fixed Till Toenshoff Till Toenshoff Till Toenshoff 29/Sep/14 14:08   19/Aug/18 13:24 16/Oct/14 00:55 3.4.5, 3.4.6, 3.5.0 3.4.7, 3.5.1, 3.6.0     2 8 0 600   OSX 10.10 (BETA3), Apple LLVM version 6.0 (clang-600.0.51) (based on LLVM 3.5svn) recordio.h defines {{htonll}} which conflicts with Apple's equally named implementation from within arpa/inet.h.


{noformat}
gcc -DHAVE_CONFIG_H -I. -I. -I. -I./include -I./tests -I./generated -Wall -Werror -g -O2 -D_GNU_SOURCE -MT zk_log.lo -MD -MP -MF .deps/zk_log.Tpo -c src/zk_log.c -fno-common -DPIC -o zk_log.o
In file included from src/recordio.c:19:
./include/recordio.h:76:9: error: expected ')'
int64_t htonll(int64_t v);
^
/usr/include/sys/_endian.h:141:25: note: expanded from macro 'htonll'
#define htonll(x) __DARWIN_OSSwapInt64(x)
^
/usr/include/libkern/_OSByteOrder.h:78:30: note: expanded from macro '__DARWIN_OSSwapInt64'
(__builtin_constant_p(x) ? __DARWIN_OSSwapConstInt64(x) : _OSSwapInt64(x))
^
./include/recordio.h:76:9: note: to match this '('
/usr/include/sys/_endian.h:141:25: note: expanded from macro 'htonll'
#define htonll(x) __DARWIN_OSSwapInt64(x)
^
/usr/include/libkern/_OSByteOrder.h:78:5: note: expanded from macro '__DARWIN_OSSwapInt64'
(__builtin_constant_p(x) ? __DARWIN_OSSwapConstInt64(x) : _OSSwapInt64(x))
^
In file included from src/recordio.c:19:
./include/recordio.h:76:9: error: conflicting types for '__builtin_constant_p'
int64_t htonll(int64_t v);
^
/usr/include/sys/_endian.h:141:25: note: expanded from macro 'htonll'
#define htonll(x) __DARWIN_OSSwapInt64(x)
^
/usr/include/libkern/_OSByteOrder.h:78:6: note: expanded from macro '__DARWIN_OSSwapInt64'
(__builtin_constant_p(x) ? __DARWIN_OSSwapConstInt64(x) : _OSSwapInt64(x))
^
./include/recordio.h:76:9: note: '__builtin_constant_p' is a builtin with type 'int ()'
/usr/include/sys/_endian.h:141:25: note: expanded from macro 'htonll'
#define htonll(x) __DARWIN_OSSwapInt64(x)
^
/usr/include/libkern/_OSByteOrder.h:78:6: note: expanded from macro '__DARWIN_OSSwapInt64'
(__builtin_constant_p(x) ? __DARWIN_OSSwapConstInt64(x) : _OSSwapInt64(x))
^
In file included from generated/zookeeper.jute.c:20:
In file included from ./generated/zookeeper.jute.h:21:
./include/recordio.h:76:9: error: expected ')'
int64_t htonll(int64_t v);
^
/usr/include/sys/_endian.h:141:25: note: expanded from macro 'htonll'
#define htonll(x) __DARWIN_OSSwapInt64(x)
^
/usr/include/libkern/_OSByteOrder.h:78:30: note: expanded from macro '__DARWIN_OSSwapInt64'
(__builtin_constant_p(x) ? __DARWIN_OSSwapConstInt64(x) : _OSSwapInt64(x))
^
./include/recordio.h:76:9: note: to match this '('
/usr/include/sys/_endian.h:141:25: note: expanded from macro 'htonll'
#define htonll(x) __DARWIN_OSSwapInt64(x)
^
/usr/include/libkern/_OSByteOrder.h:78:5: note: expanded from macro '__DARWIN_OSSwapInt64'
(__builtin_constant_p(x) ? __DARWIN_OSSwapConstInt64(x) : _OSSwapInt64(x))
^
In file included from generated/zookeeper.jute.c:20:
In file included from ./generated/zookeeper.jute.h:21:
./include/recordio.h:76:9: error: conflicting types for '__builtin_constant_p'
int64_t htonll(int64_t v);
^
/usr/include/sys/_endian.h:141:25: note: expanded from macro 'htonll'
#define htonll(x) __DARWIN_OSSwapInt64(x)
^
/usr/include/libkern/_OSByteOrder.h:78:6: note: expanded from macro '__DARWIN_OSSwapInt64'
(__builtin_constant_p(x) ? __DARWIN_OSSwapConstInt64(x) : _OSSwapInt64(x))
^
./include/recordio.h:76:9: note: '__builtin_constant_p' is a builtin with type 'int ()'
/usr/include/sys/_endian.h:141:25: note: expanded from macro 'htonll'
#define htonll(x) __DARWIN_OSSwapInt64(x)
^
/usr/include/libkern/_OSByteOrder.h:78:6: note: expanded from macro '__DARWIN_OSSwapInt64'
(__builtin_constant_p(x) ? __DARWIN_OSSwapConstInt64(x) : _OSSwapInt64(x))
^
In file included from src/zookeeper.c:27:
In file included from ./include/zookeeper.h:34:
./include/recordio.h:76:9: error: expected ')'
int64_t htonll(int64_t v);
^
/usr/include/sys/_endian.h:141:25: note: expanded from macro 'htonll'
#define htonll(x) __DARWIN_OSSwapInt64(x)
^
/usr/include/libkern/_OSByteOrder.h:78:30: note: expanded from macro '__DARWIN_OSSwapInt64'
(__builtin_constant_p(x) ? __DARWIN_OSSwapConstInt64(x) : _OSSwapInt64(x))
^
./include/recordio.h:76:9: note: to match this '('
/usr/include/sys/_endian.h:141:25: note: expanded from macro 'htonll'
#define htonll(x) __DARWIN_OSSwapInt64(x)
^
/usr/include/libkern/_OSByteOrder.h:78:5: note: expanded from macro '__DARWIN_OSSwapInt64'
(__builtin_constant_p(x) ? __DARWIN_OSSwapConstInt64(x) : _OSSwapInt64(x))
^
In file included from src/zookeeper.c:27:
In file included from ./include/zookeeper.h:34:
./include/recordio.h:76:9: error: conflicting types for '__builtin_constant_p'
int64_t htonll(int64_t v);
^
/usr/include/sys/_endian.h:141:25: note: expanded from macro 'htonll'
#define htonll(x) __DARWIN_OSSwapInt64(x)
^
/usr/include/libkern/_OSByteOrder.h:78:6: note: expanded from macro '__DARWIN_OSSwapInt64'
(__builtin_constant_p(x) ? __DARWIN_OSSwapConstInt64(x) : _OSSwapInt64(x))
^
./include/recordio.h:76:9: note: '__builtin_constant_p' is a builtin with type 'int ()'
/usr/include/sys/_endian.h:141:25: note: expanded from macro 'htonll'
#define htonll(x) __DARWIN_OSSwapInt64(x)
^
/usr/include/libkern/_OSByteOrder.h:78:6: note: expanded from macro '__DARWIN_OSSwapInt64'
(__builtin_constant_p(x) ? __DARWIN_OSSwapConstInt64(x) : _OSSwapInt64(x))
^
In file included from src/zk_hashtable.c:19:
In file included from src/zk_hashtable.h:22:
In file included from ./include/zookeeper.h:34:
./include/recordio.h:76:9: error: expected ')'
int64_t htonll(int64_t v);
^
/usr/include/sys/_endian.h:141:25: note: expanded from macro 'htonll'
#define htonll(x) __DARWIN_OSSwapInt64(x)
^
/usr/include/libkern/_OSByteOrder.h:78:30: note: expanded from macro '__DARWIN_OSSwapInt64'
(__builtin_constant_p(x) ? __DARWIN_OSSwapConstInt64(x) : _OSSwapInt64(x))
^
./include/recordio.h:76:9: note: to match this '('
/usr/include/sys/_endian.h:141:25: note: expanded from macro 'htonll'
#define htonll(x) __DARWIN_OSSwapInt64(x)
^
/usr/include/libkern/_OSByteOrder.h:78:5: note: expanded from macro '__DARWIN_OSSwapInt64'
(__builtin_constant_p(x) ? __DARWIN_OSSwapConstInt64(x) : _OSSwapInt64(x))
^
In file included from src/zk_hashtable.c:19:
In file included from src/zk_hashtable.h:22:
In file included from ./include/zookeeper.h:34:
./include/recordio.h:76:9: error: conflicting types for '__builtin_constant_p'
int64_t htonll(int64_t v);
^
/usr/include/sys/_endian.h:141:25: note: expanded from macro 'htonll'
#define htonll(x) __DARWIN_OSSwapInt64(x)
^
/usr/include/libkern/_OSByteOrder.h:78:6: note: expanded from macro '__DARWIN_OSSwapInt64'
(__builtin_constant_p(x) ? __DARWIN_OSSwapConstInt64(x) : _OSSwapInt64(x))
^
./include/recordio.h:76:9: note: '__builtin_constant_p' is a builtin with type 'int ()'
/usr/include/sys/_endian.h:141:25: note: expanded from macro 'htonll'
#define htonll(x) __DARWIN_OSSwapInt64(x)
^
/usr/include/libkern/_OSByteOrder.h:78:6: note: expanded from macro '__DARWIN_OSSwapInt64'
(__builtin_constant_p(x) ? __DARWIN_OSSwapConstInt64(x) : _OSSwapInt64(x))
^
src/recordio.c:83:9: error: expected ')'
int64_t htonll(int64_t v)
^
/usr/include/sys/_endian.h:141:25: note: expanded from macro 'htonll'
#define htonll(x) __DARWIN_OSSwapInt64(x)
^
/usr/include/libkern/_OSByteOrder.h:78:30: note: expanded from macro '__DARWIN_OSSwapInt64'
(__builtin_constant_p(x) ? __DARWIN_OSSwapConstInt64(x) : _OSSwapInt64(x))
^
src/recordio.c:83:9: note: to match this '('
/usr/include/sys/_endian.h:141:25: note: expanded from macro 'htonll'
#define htonll(x) __DARWIN_OSSwapInt64(x)
^
/usr/include/libkern/_OSByteOrder.h:78:5: note: expanded from macro '__DARWIN_OSSwapInt64'
(__builtin_constant_p(x) ? __DARWIN_OSSwapConstInt64(x) : _OSSwapInt64(x))
^
src/recordio.c:83:9: error: conflicting types for '__builtin_constant_p'
int64_t htonll(int64_t v)
^
/usr/include/sys/_endian.h:141:25: note: expanded from macro 'htonll'
#define htonll(x) __DARWIN_OSSwapInt64(x)
^
/usr/include/libkern/_OSByteOrder.h:78:6: note: expanded from macro '__DARWIN_OSSwapInt64'
(__builtin_constant_p(x) ? __DARWIN_OSSwapConstInt64(x) : _OSSwapInt64(x))
^
./include/recordio.h:76:9: note: '__builtin_constant_p' is a builtin with type 'int ()'
int64_t htonll(int64_t v);
^
/usr/include/sys/_endian.h:141:25: note: expanded from macro 'htonll'
#define htonll(x) __DARWIN_OSSwapInt64(x)
^
/usr/include/libkern/_OSByteOrder.h:78:6: note: expanded from macro '__DARWIN_OSSwapInt64'
(__builtin_constant_p(x) ? __DARWIN_OSSwapConstInt64(x) : _OSSwapInt64(x))
^
src/recordio.c:83:9: error: definition of builtin function '__builtin_constant_p'
int64_t htonll(int64_t v)
^
/usr/include/sys/_endian.h:141:25: note: expanded from macro 'htonll'
#define htonll(x) __DARWIN_OSSwapInt64(x)
^
/usr/include/libkern/_OSByteOrder.h:78:6: note: expanded from macro '__DARWIN_OSSwapInt64'
(__builtin_constant_p(x) ? __DARWIN_OSSwapConstInt64(x) : _OSSwapInt64(x))
^
In file included from src/zk_log.c:23:
In file included from ./include/zookeeper_log.h:22:
In file included from ./include/zookeeper.h:34:
./include/recordio.h:76:9: error: expected ')'
int64_t htonll(int64_t v);
^
/usr/include/sys/_endian.h:141:25: note: expanded from macro 'htonll'
#define htonll(x) __DARWIN_OSSwapInt64(x)
^
/usr/include/libkern/_OSByteOrder.h:78:30: note: expanded from macro '__DARWIN_OSSwapInt64'
(__builtin_constant_p(x) ? __DARWIN_OSSwapConstInt64(x) : _OSSwapInt64(x))
^
./include/recordio.h:76:9: note: to match this '('
/usr/include/sys/_endian.h:141:25: note: expanded from macro 'htonll'
#define htonll(x) __DARWIN_OSSwapInt64(x)
^
/usr/include/libkern/_OSByteOrder.h:78:5: note: expanded from macro '__DARWIN_OSSwapInt64'
(__builtin_constant_p(x) ? __DARWIN_OSSwapConstInt64(x) : _OSSwapInt64(x))
^
In file included from src/zk_log.c:23:
In file included from ./include/zookeeper_log.h:22:
In file included from ./include/zookeeper.h:34:
./include/recordio.h:76:9: error: conflicting types for '__builtin_constant_p'
int64_t htonll(int64_t v);
^
/usr/include/sys/_endian.h:141:25: note: expanded from macro 'htonll'
#define htonll(x) __DARWIN_OSSwapInt64(x)
^
/usr/include/libkern/_OSByteOrder.h:78:6: note: expanded from macro '__DARWIN_OSSwapInt64'
(__builtin_constant_p(x) ? __DARWIN_OSSwapConstInt64(x) : _OSSwapInt64(x))
^
./include/recordio.h:76:9: note: '__builtin_constant_p' is a builtin with type 'int ()'
/usr/include/sys/_endian.h:141:25: note: expanded from macro 'htonll'
#define htonll(x) __DARWIN_OSSwapInt64(x)
^
/usr/include/libkern/_OSByteOrder.h:78:6: note: expanded from macro '__DARWIN_OSSwapInt64'
(__builtin_constant_p(x) ? __DARWIN_OSSwapConstInt64(x) : _OSSwapInt64(x))
^
2 errors generated.
5 errors generated.
2 errors generated.
make[5]: *** [recordio.lo] Error 1
make[5]: *** Waiting for unfinished jobs....
2 errors generated.
make[5]: *** [zookeeper.jute.lo] Error 1
make[5]: *** [zk_hashtable.lo] Error 1
make[5]: *** [zk_log.lo] Error 1
2 errors generated.
make[5]: *** [zookeeper.lo] Error 1
make[4]: *** [all] Error 2
make[3]: *** [zookeeper-3.4.5/src/c/libzookeeper_mt.la] Error 2
make[3]: *** Waiting for unfinished jobs....
ln -fs libleveldb.dylib.1.4 libleveldb.dylib
ln -fs libleveldb.dylib.1.4 libleveldb.dylib.1
make[2]: *** [all-recursive] Error 1
make[1]: *** [all] Error 2
make: *** [all-recursive] Error 1
{noformat}
100% 100% 600 0 pull-request-available 9223372036854775807 No Perforce job exists for this issue. 3 9223372036854775807
3 years, 22 weeks, 3 days ago 0|i20ljb:
ZooKeeper ZOOKEEPER-2048

Ability to support alphabetic characters in the version string

Improvement Open Major Unresolved Unassigned Chunjun Xiao Chunjun Xiao 26/Sep/14 03:10   26/Sep/14 03:11   3.4.5   build   0 1   Following ZOOKEEPER-1598, could we further enhance ZK to support alphabetic characters in the version string?
E.g., zookeeper-3.4.5.ABC.1.2.3.4-1.jar.
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
5 years, 25 weeks, 6 days ago 0|i20igf:
ZooKeeper ZOOKEEPER-2047

ZOOKEEPER-1833 testTruncationNullLog fails on windows

Sub-task Resolved Major Fixed Flavio Paiva Junqueira Flavio Paiva Junqueira Flavio Paiva Junqueira 25/Sep/14 13:35   27/Sep/14 07:16 27/Sep/14 02:32 3.4.6 3.4.7, 3.5.1, 3.6.0 tests   0 3   The calls to delete the log file on windows are failing, so the test ends up failing. The fix is to close the db before deleting. 9223372036854775807 No Perforce job exists for this issue. 2 9223372036854775807
5 years, 25 weeks, 5 days ago 0|i20hk7:
ZooKeeper ZOOKEEPER-2046

Compile zookeeper by JDK 7 in default

Bug Resolved Major Duplicate Unassigned Guo Ruijing Guo Ruijing 23/Sep/14 02:43   02/Dec/14 16:22 02/Dec/14 16:22 3.5.0   build   0 7   Currently, zookeeper is compiled by JDK 5 in default as

<property name="javac.target" value="1.5" />
<property name="javac.source" value="1.5" />

we may change it to JDK 7 in default
build 9223372036854775807 No Perforce job exists for this issue. 2 9223372036854775807
5 years, 16 weeks, 2 days ago 0|i20dqn:
ZooKeeper ZOOKEEPER-2045

ConnectStringParser public api didn't handle null connect string

Bug Patch Available Minor Unresolved Hongchao Deng Hongchao Deng Hongchao Deng 22/Sep/14 19:17   22/Sep/14 20:22   3.5.0       0 1   {code}
public final class ConnectStringParser {
...
public ConnectStringParser(String connectString) {
...
{code}
ConnectStringParser is a public api. Besides that, both ZooKeeper constructor and ZooKeeper#updateServerList used it.

However, it doesn't handle a null connectString. It doesn't help that much to see a NPE showing up. So I add a check to the constructor.
9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
5 years, 26 weeks, 3 days ago 0|i20d87:
ZooKeeper ZOOKEEPER-2044

CancelledKeyException in zookeeper branch-3.4

Bug Closed Minor Fixed Michael Han shamjith antholi shamjith antholi 22/Sep/14 06:17   19/Sep/19 02:36 31/Jan/17 11:40 3.4.6 3.4.10 server   3 21 0 600   Red Hat Enterprise Linux Server release 6.2 I am getting cancelled key exception in zookeeper (version 3.4.5). Please see the log below. When this error is thrown, the connected solr shard is going down by giving the error "Failed to index metadata in Solr,StackTrace=SolrError: HTTP status 503.Reason: {"responseHeader":{"status":503,"QTime":204},"error":{"msg":"ClusterState says we are the leader, but locally we don't think so","code":503" and ultimately the current activity is going down. Could you please give a solution for this ?


Zookeper log
----------------------------------------------------------
2014-09-16 02:58:47,799 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@832] - Client attempting to renew session 0x24868e7ca980003 at /172.22.0.5:58587
2014-09-16 02:58:47,800 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:Learner@107] - Revalidating client: 0x24868e7ca980003
2014-09-16 02:58:47,802 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:ZooKeeperServer@588] - Invalid session 0x24868e7ca980003 for client /172.22.0.5:58587, probably expired
2014-09-16 02:58:47,803 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1001] - Closed socket connection for client /172.22.0.5:58587 which had sessionid 0x24868e7ca980003
2014-09-16 02:58:47,810 [myid:1] - ERROR [CommitProcessor:1:NIOServerCnxn@180] - Unexpected Exception:
java.nio.channels.CancelledKeyException
at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:55)
at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:59)
at org.apache.zookeeper.server.NIOServerCnxn.sendBuffer(NIOServerCnxn.java:153)
at org.apache.zookeeper.server.NIOServerCnxn.sendResponse(NIOServerCnxn.java:1076)
at org.apache.zookeeper.server.NIOServerCnxn.process(NIOServerCnxn.java:1113)
at org.apache.zookeeper.server.DataTree.setWatches(DataTree.java:1327)
at org.apache.zookeeper.server.ZKDatabase.setWatches(ZKDatabase.java:384)
at org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:304)
at org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:74)


100% 100% 600 0 pull-request-available 9223372036854775807 No Perforce job exists for this issue. 2 9223372036854775807
3 years, 6 weeks, 4 days ago 0|i20brr:
ZooKeeper ZOOKEEPER-2043

Too many connections,maxClientCnxns don't close

Bug Open Major Unresolved Unassigned hikin hikin 22/Sep/14 05:15   17/Sep/19 08:54           0 3   org.apache.zookeeper.server.NIOServerCnxn
void doIO(SelectionKey k) throws InterruptedException {
try {
if (isSocketOpen() == false) {
LOG.warn("trying to do i/o on a null socket for session:0x"
+ Long.toHexString(sessionId));

return;
}
public void close() {
if (!factory.removeCnxn(this)) {
return;
}
If the socket suddenly broken, do not have the right to clean up the connection, this one line of code that caused a lot of links, eventually exceed the maximum maxClientCnxns, cause the client end connections do not go up.



9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
26 weeks, 2 days ago 0|i20bo7:
ZooKeeper ZOOKEEPER-2042

zkServer.sh does not work properly on Solaris

Bug Resolved Minor Duplicate Chris Nauroth John Lindwall John Lindwall 17/Sep/14 13:44   01/May/15 01:56 01/May/15 01:56 3.4.6   scripts   1 3   Solaris 5.11 There are two issues in the zkServer.sh script that make it not work properly out of the box on Solaris.

1. The bin/zkServer.sh script uses plain "echo" in all instances but one: when writing the pid to the pid file. In that instance it uses "/bin/echo".

The "/bin/echo" command on Solaris does not understand the "-n" parameter and interprets it as a literal string, so the "-n" gets written into the pid file along with the pid. This causes the "stop" command to fail.

2. The /bin/grep command in Solaris does not understand special character classes like "[[:space:]]". You must use the alternate posix version of grep as found in /usr/xpg4/bin/grep for this to work. If the script cannot be made completely generic then at least we should document the need to use the posix grep implementation on Solaris.
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
4 years, 46 weeks, 6 days ago 0|i20647:
ZooKeeper ZOOKEEPER-2041

maxClientCnxns should limit connections for each process rather than ip

Improvement Open Major Unresolved Unassigned chendihao chendihao 16/Sep/14 22:10   16/Sep/14 22:10   3.4.4   server   1 1   Recently I learned more about maxClientCnxns configuration and read the read the code of implement. I know now it's the limitation of the number of connection from the same ip.

But actually we may run multiple process in the same server. And if one process excesses the limitation of maxClientCnxns, all the ZooKeeper clients will fail to connect with ZooKeeper cluster. Can we fix that to make this limitation for each process?

Any suggestion is welcome.
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
5 years, 27 weeks, 1 day ago 0|i204zb:
ZooKeeper ZOOKEEPER-2040

Server to log underlying cause of SASL connection problems

Improvement Closed Major Fixed Steve Loughran Steve Loughran Steve Loughran 16/Sep/14 12:05   21/Jul/16 16:18 11/Sep/15 02:06 3.4.6 3.4.7, 3.5.2, 3.6.0 server   0 7   When you have SASL connectivity problems, you spend time staring at logs —ideally logs with stack traces.

ZK server can help here by including the stack traces when there is a SASL auth problem, rather than just giving the text of the exception.
9223372036854775807 No Perforce job exists for this issue. 2 9223372036854775807
4 years, 27 weeks, 6 days ago
Reviewed
0|i203vz:
ZooKeeper ZOOKEEPER-2039

Jute compareBytes incorrect comparison index

Bug Resolved Minor Fixed Ian Dimayuga Ian Dimayuga Ian Dimayuga 15/Sep/14 22:34   28/Sep/14 05:48 18/Sep/14 11:51 3.4.6, 3.5.0 3.4.7, 3.5.1 jute   0 5 86400 86400 0% The Jute utility's naïve byte-array comparison compares b1[off1+i] with b2[off2+1]. (A literal 1, not the variable i)

It should be off2+i, in parallel with the other operand.
0% 0% 86400 86400 9223372036854775807 No Perforce job exists for this issue. 2 9223372036854775807
5 years, 25 weeks, 4 days ago 0|i202qn:
ZooKeeper ZOOKEEPER-2038

Coding error in recipes/lock/src/c/src/zoo_lock.c

Bug Open Minor Unresolved Unassigned YuQing YuQing 15/Sep/14 22:23   15/Sep/14 22:23   3.4.6   recipes   0 1   arch linux 64bit In fuction child_floor(), strcmp() is used to compare the whole string.
But there exists conditions a sorted_data looks like ("x-000-00", "x-222-01", "x-111-02"), and now "x-222-01" is calling child_floor() to get a predecessor for watching, so the current logic will return "x-111-02" instead of the correct "x-000-00".
Use a strcmp() == 0 and a break statement should solve this problem.
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
5 years, 27 weeks, 2 days ago 0|i202pz:
ZooKeeper ZOOKEEPER-2037

ZOOKEEPER-2016 ZooKeeper methods to wait on client connection (re)establishment.

Sub-task Resolved Major Won't Fix Hongchao Deng Hongchao Deng Hongchao Deng 15/Sep/14 17:00   27/Jan/15 23:32 27/Jan/15 23:32 3.5.0   java client   0 3   When a ZooKeeper object is created and returned, it is not guaranteed that connection to a server is established.

Usually, a wait/signal pattern is used for the ZK watcher
{code}
latch = new CountDownLatch(1)
zk = new ZooKeeper(..., new Watcher() {
override void process(WatchedEvent event) {
if (event.type = SyncConnected) {
latch.countDown()
}
}
},...)

latch.await();
// connection has been established. do something with zk.
{code}

There are two disadvantages:
1. The latch object isn't being garbage-collected. Because the watcher keeps monitoring all kinds of events.
2. With the introduction of dynamic reconfig, client move to other servers on needed and this latch method doesn't work so well.

Here I propose to add (both sync and async) wait methods to act as latch for connection establish such that it becomes much easier to manage and work around:
{code}
zk = new ZooKeeper(...)
zk.waitUntilConnected()
{code}
9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
5 years, 8 weeks, 1 day ago 0|i202dr:
ZooKeeper ZOOKEEPER-2036

Client which is not authorized able to access the Secure Data which is created by the Secure Client

Bug Resolved Blocker Not A Problem Unassigned Brahma Reddy Battula Brahma Reddy Battula 15/Sep/14 04:08   28/Oct/14 09:24 28/Oct/14 09:24 3.4.5   server   0 2   *{color:blue}Scenario:{color}*

Started the Secure ZK Cluster.
Logged with Secure ZK Client(by passing valid jaas.conf) and created the Znodes

Now logged in to same secure cluster using unsecure ZKClient (without jaas.conf) to same Cluster and able to access the data which is created by the Secured Client..

*{color:blue}Secured Client{color}:(which is created the Znodes)*

2014-09-15 13:40:56,288 [myid:] - INFO [main-SendThread(localhost:2181):ZooKeeperSaslClient$1@285] - Client will use GSSAPI as SASL mechanism.
2014-09-15 13:40:56,296 [myid:] - INFO [Thread-1:Login@301] - TGT valid starting at: Mon Sep 15 13:40:56 IST 2014
2014-09-15 13:40:56,296 [myid:] - INFO [Thread-1:Login@302] - TGT expires: Tue Sep 16 13:40:56 IST 2014
2014-09-15 13:40:56,296 [myid:] - INFO [Thread-1:Login$1@181] - TGT refresh sleeping until: Tue Sep 16 09:36:04 IST 2014
2014-09-15 13:40:56,302 [myid:] - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread@1000] - Opening socket connection to server localhost/0:0:0:0:0:0:0:1:2181. Will attempt to SASL-authenticate using Login Context section 'Client'
2014-09-15 13:40:56,308 [myid:] - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread@855] - Socket connection established to localhost/0:0:0:0:0:0:0:1:2181, initiating session
2014-09-15 13:40:56,344 [myid:] - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread@1260] - Session establishment complete on server localhost/0:0:0:0:0:0:0:1:2181, sessionid = 0x1486856657e0016, negotiated timeout = 30000

WATCHER::

WatchedEvent state:SyncConnected type:None path:null

WATCHER::

WatchedEvent state: *{color:red}SaslAuthenticated{color}* type:None path:null

[zk: localhost:2181(CONNECTED) 1] create -s /tmp-seq 'sd:er:'
Created /tmp-seq0000000003
[zk: localhost:2181(CONNECTED) 2] create -s /tmp-seq 'sd:er:'
Created /tmp-seq0000000004
[zk: localhost:2181(CONNECTED) 0] ls /
[tmp-seq0000000004, tmp-seq0000000003, hadoop, hadoop-ha, tmp-seq0000000002, zookeeper]

*{color:blue}UnSecured Client{color}:(which is Accesing Znodes)*
Welcome to ZooKeeper!
2014-09-15 13:00:30,440 [myid:] - WARN [main-SendThread(localhost:2181):ClientCnxn$SendThread@982] - SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: '/home/****/zookeeper/conf/jaas.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.
014-09-15 13:00:30,441 [myid:] - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread@1000] - Opening socket connection to server localhost/127.0.0.1:2181
WatchedEvent state: *{color:red}AuthFailed{color}* type:None path:null
JLine support is enabled
2014-09-15 13:00:30,451 [myid:] - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread@855] - Socket connection established to localhost/127.0.0.1:2181, initiating session
[zk: localhost:2181(CONNECTING) 0] 2014-09-15 13:00:30,488 [myid:] - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread@1260] - Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x348685662250005, negotiated timeout = 30000

WATCHER::

WatchedEvent state:SyncConnected type:None path:null

[zk: localhost:2181(CONNECTED) 0] ls /
[tmp-seq0000000004, tmp-seq0000000003, hadoop, hadoop-ha, tmp-seq0000000002, zookeeper]
[zk: localhost:2181(CONNECTED) 1] get /tmp-seq000000000

tmp-seq0000000004 tmp-seq0000000003 tmp-seq0000000002
[zk: localhost:2181(CONNECTED) 1] get /tmp-seq0000000002
''
cZxid = 0x100000040
ctime = Mon Sep 15 12:51:50 IST 2014
mZxid = 0x100000040
mtime = Mon Sep 15 12:51:50 IST 2014
pZxid = 0x100000040
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 2
numChildren = 0


9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
5 years, 21 weeks, 2 days ago 0|i201br:
ZooKeeper ZOOKEEPER-2035

diagnostics on SASL connection problems doesn't match error strings sent back

Bug Open Minor Unresolved Unassigned Steve Loughran Steve Loughran 13/Sep/14 13:33   15/Dec/15 12:47   3.4.6       0 3   Java 1.7.0.67 on OS/X The diagnostics code in {{ZooKeeperSaslClient.createSaslToken()}} which looks for a {{"("Mechanism level: Server not found in Kerberos database (7) - UNKNOWN_SERVER)"}} error string isn't finding a match ... the text now appears to be {{(Mechanism level: Server not found in Kerberos database (7) - Server not found in Kerberos database)}} 9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
5 years, 27 weeks, 5 days ago 0|i200dz:
ZooKeeper ZOOKEEPER-2034

StringIndexOutOfBoundsException in createSaslServer

Bug Resolved Minor Not A Problem Unassigned Steve Loughran Steve Loughran 13/Sep/14 13:09   13/Sep/14 13:11 13/Sep/14 13:11 3.4.6       0 1   I'm seeing {{StringIndexOutOfBoundsException}} in {{createSaslServer}}, where my test kerberos code is (presumably) is not correctly set up.

Looking at the comments, it hints that the problem is my principals are called {{zookeeper@EXAMPLE.COM}}, which doesn't match the pattern {{principal/host@realm}}
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
5 years, 27 weeks, 5 days ago 0|i200dj:
ZooKeeper ZOOKEEPER-2033

zookeeper follower fails to start after a restart immediately following a new epoch

Bug Resolved Major Fixed Asad Saeed Asad Saeed Asad Saeed 10/Sep/14 14:14   03/Sep/15 20:06 03/Sep/15 19:00 3.4.6 3.4.7 quorum   0 5   The following issue was seen when adding a new node to a zookeeper cluster.

Reproduction steps
1. Create a 2 node ensemble. Write some keys.
2. Add another node to the ensemble, by modifying the config. Restarting entire cluster.
3. Restart the new node before writing any new keys.

What occurs is that the new node gets a SNAP from the newly elected leader, since it is too far behind. The zxid for this snapshot is from the new epoch but that is not in the committed log cache.

On restart of this new node. The follower sends the new epoch zxid. The leader looks at it's maxCommitted logs, and sees that it is not the newest epoch, and therefore sends a TRUNC.

The follower sees the TRUNC but it only has a snapshot, so it cannot truncate!
9223372036854775807 No Perforce job exists for this issue. 2 9223372036854775807
4 years, 29 weeks ago 0|i1zw5j:
ZooKeeper ZOOKEEPER-2032

ReconfigBackupTest didn't clean up resources.

Test Resolved Minor Fixed Hongchao Deng Hongchao Deng Hongchao Deng 09/Sep/14 19:33   17/Sep/14 10:11 10/Sep/14 03:03 3.5.0 3.5.1 tests   0 4   9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
5 years, 27 weeks, 1 day ago 0|i1zuun:
ZooKeeper ZOOKEEPER-2031

Support tagging a QuorumServer

Improvement Patch Available Major Unresolved some one some one some one 09/Sep/14 16:43   05/Feb/20 07:11     3.7.0, 3.5.8 server   0 5   Currently ZooKeeper only allows using the server id which is an integer for identifying servers. For my (unavoidable) use case, there may be concurrent dynamic removes and adds of servers which may eventually have id collisions. When this occurs, there is no good way to determine if the server (given an id collision) that we want to remove is the right server.

To support my use case, I propose that we add a tag field to the server string.
For my specific use case, this tag field will be used to store a uuid as a string.
So for example:
server.1=127.0.0.1:1234:1236:participant;0.0.0.0:1237;743b9d23-85cb-45b1-8949-930fdabb21f0

9223372036854775807 No Perforce job exists for this issue. 2 9223372036854775807
1 year, 45 weeks ago 0|i1zuo7:
ZooKeeper ZOOKEEPER-2030

dynamicConfigFile should have an absolute path, not a relative path, to the dynamic configuration file

Bug Resolved Minor Fixed Alexander Shraer Alexander Shraer Alexander Shraer 07/Sep/14 00:42   17/Sep/14 07:11 17/Sep/14 01:36 3.5.0 3.5.1, 3.6.0 server   0 5   a relative path doesn't seem like a good idea since it will work only if we start the server from the same directory as we did previously. 9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
5 years, 27 weeks, 1 day ago 0|i1zrpz:
ZooKeeper ZOOKEEPER-2029

Leader.LearnerCnxAcceptor should handle exceptions in run()

Bug Resolved Minor Fixed Rakesh Radhakrishnan Asad Saeed Asad Saeed 05/Sep/14 19:21   08/May/15 14:55 11/Apr/15 14:49 3.4.6 3.5.1, 3.6.0 quorum   0 5   Leader.LearnerCnxAcceptor swallows exceptions and shuts itself down. It should instead crash the Leader. 9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
4 years, 49 weeks, 5 days ago 0|i1zqzr:
ZooKeeper ZOOKEEPER-2028

TestClient#testAuth aborts because ASSERT throws exception again in destructor when there is active exception already

Bug Patch Available Minor Unresolved Qiang Tian Qiang Tian Qiang Tian 05/Sep/14 03:20   19/Jul/17 16:25   3.4.6   tests   0 3   linux Hi Guys,
the testcase consistently fails if debug is turned on(set zoo_set_debug_level(ZOO_LOG_LEVEL_DEBUG) in TestDriver.cc); if debug is OFF, it fails for the first time, subsequent runs succeed.

can someone help take a look?
thanks!

below is related info:

1. screen output
{quote}
[exec] Zookeeper_simpleSystem::testPing : elapsed 17200 : OK
[exec] Zookeeper_simpleSystem::testAcl : elapsed 1014 : OK
[exec] Zookeeper_simpleSystem::testChroot : elapsed 3041 : OK
[exec] terminate called after throwing an instance of 'CppUnit::Exception'
[exec] what(): equality assertion failed
[exec] - Expected: 0
[exec] - Actual : -116
[exec]
[exec] make: *** [run-check] Aborted (core dumped)
[exec] Zookeeper_simpleSystem::testAuth
{quote}

2. last lines in zk server log:
{quote}
2014-09-04 21:13:57,711 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:22181:ZooKeeperServer@868] - Client attempting to establish new session at /127.0.0.1:34992
2014-09-04 21:13:57,714 [myid:] - INFO [SyncThread:0:ZooKeeperServer@617] - Established session 0x14844044d96000a with negotiated timeout 10000 for client /127.0.0.1:34992
2014-09-04 21:14:01,039 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:22181:ZooKeeperServer@892] - got auth packet /127.0.0.1:34992
2014-09-04 21:14:01,747 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:22181:ZooKeeperServer@926] - auth success /127.0.0.1:34992
2014-09-04 21:14:01,912 [myid:] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:22181:NIOServerCnxn@362] - Exception causing close of session 0x14844044d96000a due to java.io.IOException: Connection reset by peer
2014-09-04 21:14:01,914 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:22181:NIOServerCnxn@1007] - Closed socket connection for client /127.0.0.1:34992 which had sessionid 0x14844044d96000a
2014-09-04 21:14:12,000 [myid:] - INFO [SessionTracker:ZooKeeperServer@347] - Expiring session 0x14844044d96000a, timeout of 10000ms exceeded
2014-09-04 21:14:12,001 [myid:] - INFO [ProcessThread(sid:0 cport:-1)::PrepRequestProcessor@494] - Processed session termination for sessionid: 0x14844044d96000a
{quote}

3. last lines in TEST-Zookeeper_simpleSystem-mt.txt:
{quote}
2014-09-04 21:13:57,703:383481(0x7f8866c4b720):ZOO_INFO@log_env@712: Client environment:zookeeper.version=zookeeper C client 3.4.6
2014-09-04 21:13:57,703:383481(0x7f8866c4b720):ZOO_INFO@log_env@716: Client environment:host.name=localhost
2014-09-04 21:13:57,703:383481(0x7f8866c4b720):ZOO_INFO@log_env@723: Client environment:os.name=Linux
2014-09-04 21:13:57,703:383481(0x7f8866c4b720):ZOO_INFO@log_env@724: Client environment:os.arch=2.6.32-358.el6.x86_64
2014-09-04 21:13:57,703:383481(0x7f8866c4b720):ZOO_INFO@log_env@725: Client environment:os.version=#1 SMP Tue Jan 29 11:47:41 EST 2013
2014-09-04 21:13:57,703:383481(0x7f8866c4b720):ZOO_INFO@log_env@733: Client environment:user.name=tianq
2014-09-04 21:13:57,703:383481(0x7f8866c4b720):ZOO_INFO@log_env@741: Client environment:user.home=/home/tianq
2014-09-04 21:13:57,703:383481(0x7f8866c4b720):ZOO_INFO@log_env@753: Client environment:user.dir=/home/tianq/zookeeper/build/test/test-cppunit
2014-09-04 21:13:57,703:383481(0x7f8866c4b720):ZOO_INFO@zookeeper_init@786: Initiating client connection, host=127.0.0.1:22181 sessionTimeout=10000 watcher=0x42e590 sessionId=0 sessionPasswd=<null> context=0x7fff695ea9a0 flags=0
2014-09-04 21:13:57,703:383481(0x7f8866c4b720):ZOO_DEBUG@start_threads@221: starting threads...
2014-09-04 21:13:57,704:383481(0x7f8857fff700):ZOO_DEBUG@do_io@367: started IO thread
2014-09-04 21:13:57,704:383481(0x7f8857fff700):ZOO_INFO@check_events@1705: initiated connection to server [127.0.0.1:22181]
2014-09-04 21:13:57,704:383481(0x7f88667f9700):ZOO_DEBUG@do_completion@459: started completion thread
2014-09-04 21:13:57,714:383481(0x7f8857fff700):ZOO_INFO@check_events@1752: session establishment complete on server [127.0.0.1:22181], sessionId=0x14844044d96000a, negotiated timeout=10000
2014-09-04 21:13:57,714:383481(0x7f8857fff700):ZOO_DEBUG@check_events@1758: Calling a watcher for a ZOO_SESSION_EVENT and the state=ZOO_CONNECTED_STATE
2014-09-04 21:13:57,714:383481(0x7f88667f9700):ZOO_DEBUG@process_completions@2113: Calling a watcher for node [], type = -1 event=ZOO_SESSION_EVENT
2014-09-04 21:13:58,704:383481(0x7f8866c4b720):ZOO_DEBUG@send_last_auth_info@1353: Sending auth info request to 127.0.0.1:22181
{quote}

If I understand correctly, it fails because assert expected 0, but looking at the testcase log, "Sending auth info request to .." appears for the first time, so it should correspond to the first zoo_add_auth call in testAuth. but its expected value is ZBADARGUMENTS...?
9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
2 years, 35 weeks, 1 day ago 0|i1zpqn:
ZooKeeper ZOOKEEPER-2027

Dynamic quorum weight shifting between datacenters for follow the sun operations

Improvement Open Minor Unresolved Unassigned Hari Sekhon Hari Sekhon 04/Sep/14 05:31   04/Sep/14 05:31           0 1   Relating to ZOOKEEPER-107 which has just added dynamic membership configuration, I'd like to propose a simplified one command weight shifting between datacenters.

This will allow for a globalized quorum where the primary datacenter in a follow the sun model gets highest weighting and can achieve low latency quorum without going over the wan. Therefore it's workload can be prioritized during it's business hours.

WANdisco has this capability which is used for it's globalized HDFS namespace control.

Obviously the current quorum majority DC must be accessible in order to initiate the quorum failover in such as scenario and the follow the sun nature of this idea also requires this to be scheduler friendly to automatically follow the sun and shift quorum majority voting several times in a 24-hour period. A single cronned zookeeper command on any zookeeper server should trigger the global coordination and handover of quorum majority to the designated DC.
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
5 years, 29 weeks ago 0|i1zocn:
ZooKeeper ZOOKEEPER-2026

Startup order in ServerCnxnFactory-ies is wrong

Bug Resolved Minor Fixed Stevo Slavić Stevo Slavić Stevo Slavić 02/Sep/14 03:21   29/Sep/14 06:31 28/Sep/14 13:23 3.4.6, 3.5.0 3.4.7, 3.5.1, 3.6.0 jmx, server   0 5   {{NIOServerCnxnFactory}} and {{NettyServerCnxnFactory}} {{startup}} method implementations are binding {{ZooKeeperServer}} too late, so in {{ZooKeeperServer}} in its startup can fail to register appropriate JMX MBean.

See [this|http://mail-archives.apache.org/mod_mbox/zookeeper-user/201409.mbox/%3CCAAUywg9-ad3oWfqRWahB9PyBEbg6%2Bd%3DDyj5PAUU7A%3Dm9wRncaw%40mail.gmail.com%3E] post on ZK user mailing list for more details.
9223372036854775807 No Perforce job exists for this issue. 3 9223372036854775807
5 years, 25 weeks, 3 days ago jmx 0|i1zkfr:
ZooKeeper ZOOKEEPER-2025

Single-node ejection caused apparent reconnection storm, leading to cluster unresponsiveness

Bug Open Major Unresolved Unassigned Stephen Tyree Stephen Tyree 29/Aug/14 16:21   16/Oct/14 13:40   3.4.5   c client, server   1 7   Description will be included in an attached PDF.

The two main questions we have are:
1: What would be the cause of the "Unreasonable Length" error in our context, and how might we prevent it from occurring?
2: What can we do to prevent the reconnection storm that led to the cluster becoming unresponsive?
9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
5 years, 23 weeks ago 0|i1zi5r:
ZooKeeper ZOOKEEPER-2024

Major throughput improvement with mixed workloads

Improvement Resolved Major Fixed Kfir Lev-Ari Kfir Lev-Ari Kfir Lev-Ari 28/Aug/14 11:41   17/Oct/19 06:24 15/May/16 17:35   3.6.0 quorum, server   2 23   The patch is applied to the commit processor, and solves two problems:

1. Stalling - once the commit processor encounters a local write request, it stalls local processing of all sessions until it receives a commit of that request from the leader.
In mixed workloads, this severely hampers performance as it does not allow read-only sessions to proceed at faster speed than read-write ones.
2. Starvation - as long as there are read requests to process, older remote committed write requests are starved.
This occurs due to a bug fix (https://issues.apache.org/jira/browse/ZOOKEEPER-1505) that forces processing of local read requests before handling any committed write. The problem is only manifested under high local read load.

Our solution solves these two problems. It improves throughput in mixed workloads (in our tests, by up to 8x), and reduces latency, especially higher percentiles (i.e., slowest requests).
The main idea is to separate sessions that inherently need to stall in order to enforce order semantics, from ones that do not need to stall. To this end, we add data structures for buffering and managing pending requests of stalled sessions; these requests are moved out of the critical path to these data structures, allowing continued processing of unaffected sessions.

Please see the docs:
1) https://goo.gl/m1cINJ - includes a detailed description of the new commit processor algorithm.

2) The attached patch implements our solution, and a collection of related unit tests (https://reviews.apache.org/r/25160)

3) https://goo.gl/W0xDUP - performance results.
(See https://issues.apache.org/jira/browse/ZOOKEEPER-2023 for the corresponding new system test that produced these performance measurements)

See also https://issues.apache.org/jira/browse/ZOOKEEPER-1609
9223372036854775807 No Perforce job exists for this issue. 13 9223372036854775807
3 years, 6 weeks, 2 days ago 0|i1zg0v:
ZooKeeper ZOOKEEPER-2023

Improved system test

Test Patch Available Minor Unresolved Kfir Lev-Ari Kfir Lev-Ari Kfir Lev-Ari 28/Aug/14 07:29   11/Sep/14 05:10   3.5.0   contrib-fatjar   0 2   Adding the ability to perform a system test of mixed workloads using read-only/mixed/write-only clients.
In addition, adding few basic latency statistics.

https://reviews.apache.org/r/25217/

Just in case it'll help someone, here is an example of how to run generate load system test:
1. Checkout zookeeper-trunk
2. Go to zookeeper-trunk, run "ant jar compile-test"
3. Go to zookeeper-trunk\src\contrib\fatjar, run "ant jar"
4. Copy zookeeper-dev-fatjar.jar from zookeeper-trunk\build\contrib\fatjar to each of the machines you wish to use.
5. On each server, assuming that you've created a valid ZK config file (e.g., zk.cfg) and a dataDir, run:
5.1 java -jar zookeeper-dev-fatjar.jar server ./zk.cfg &
5.2 java -jar zookeeper-dev-fatjar.jar ic <name of this server>:<its client port> <name of this server>:<its client port> /sysTest &
6. And finally, in order to run the test (from some machine), execute the command:
java -jar zookeeper-dev-fatjar.jar generateLoad <name of one of the servers>:<its client port> /sysTest <number of servers> <number of read-only clients> <number of mixed workload clients> <number of write-only clients>
Note that "/sysTest" is the same name that we used in 5.2.

You'll see "Preferred List is empty" message, and after few seconds you should get notifications of "Accepted connection from Socket[....".
Afterwards, just set the percentage of the mixed workload clients by entering "percentage <number>" and the test will start.

Some explanation regarding the new output (which is printed every 6 seconds, and is reset every time you enter a new percentage).
Interval: <interval number> <time>
Test info: <number of RO clients>xRO <number of mixed workload clients>x<their write percentage>%W <number of write only clients>xWO, percentiles [0.5, 0.9, 0.95, 0.99]
Throughput: <current interval throughput> | <minimum throughput until now> <average throughput until now> <maximum throughput until now>
Read latency: interval [<interval's read latency values according to the percentiles>], total [<read latency values until now, according to the percentiles>]
Write latency: interval [interval's write latency values according to the percentiles], total [<write latency values until now, according to the percentiles>]

Note that the throughput is requests per second, and latency is in ms. In addition, if you perform a read only test / write only test, you won't see the printout of write / read latency.


9223372036854775807 No Perforce job exists for this issue. 2 9223372036854775807
5 years, 28 weeks ago 0|i1zfq7:
ZooKeeper ZOOKEEPER-2022

The log of sessionTimeout is inaccurate

Improvement Open Minor Unresolved Unassigned chendihao chendihao 28/Aug/14 04:29   28/Aug/14 06:10       java client   0 2   When the client constructs ZooKeeper object, it will record the basic info in log. But its sessionTimeout may be inaccurate if it's not equal to negotiationSessionTimeout. Can we change the description of this info?

{code:java}
public ZooKeeper(String connectString, int sessionTimeout, Watcher watcher)
throws IOException
{
LOG.info("Initiating client connection, connectString=" + connectString
+ " sessionTimeout=" + sessionTimeout + " watcher=" + watcher);

watchManager.defaultWatcher = watcher;
cnxn = new ClientCnxn(connectString, sessionTimeout, this, watchManager);
cnxn.start();
}
{code}
9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
5 years, 30 weeks ago 0|i1zfk7:
ZooKeeper ZOOKEEPER-2021

ZKserver.cmd fails usng the config param

Bug Open Major Unresolved Unassigned Alvaro Gareppe Alvaro Gareppe 26/Aug/14 09:22   26/Aug/14 09:22   3.4.6   server   0 1   Windows When usgin the command like this:

zkServer.cmd zoo.cfg
we get this error:
ERROR: Invalid arguments, exiting abnormally
java.lang.NumberFormatException: For input string :"C:\Development\zookeeperserver-3.4.6\bin\..\conf\zoo.cfg"

Patch (Workaround):
change the code in zkServer.cfg to:

setlocal
call "%~dp0zkEnv.cmd"

set ZOOCFG=%ZOOCFGDIR%\%1

set ZOOMAIN=org.apache.zookeeper.server.quorum.QuorumPeerMain
echo on
java "-Dzookeeper.log.dir=%ZOO_LOG_DIR%" "-Dzookeeper.root.logger=%ZOO_LOG4J_PROP%" -cp "%CLASSPATH%" %ZOOMAIN% "%ZOOCFG%"

endlocal




9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
5 years, 30 weeks, 2 days ago 0|i1zc9z:
ZooKeeper ZOOKEEPER-2020

ZOOKEEPER-1549 Change TRUNC to SNAP in sync phase for safety guarantee

Sub-task Open Major Unresolved Unassigned Hongchao Deng Hongchao Deng 24/Aug/14 15:04   05/Feb/20 07:16   3.5.0 3.7.0, 3.5.8 quorum   0 2   ZOOKEEPER-1549 discusses the problem:

"When the leader started, it will apply every txn in its txnlog (incl. uncommitted ones) into its in-memory data tree"

I didn't any solution so far solved this problem in 3.5.x. Since this affects only TRUNC part -- only old leader that needs TRUNC applies uncommitted txns, a simple fix would be change current TRUNC logic to SNAP. This isn't hard to implement, but guarantees safety.

Ideally, we will solve the whole problem by untangling all compatibility issues and fixing the protocol.
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
5 years, 30 weeks, 4 days ago 0|i1za0n:
ZooKeeper ZOOKEEPER-2019

Unhandled exception when setting invalid limits data in /zookeeper/quota/some/path/zookeeper_limits

Bug Patch Available Major Unresolved Raúl Gutiérrez Segalés Raúl Gutiérrez Segalés Raúl Gutiérrez Segalés 22/Aug/14 19:23   05/Feb/20 07:11     3.7.0, 3.5.8 server   0 5   If you have quotas properly set for a given path, i.e.:

{noformat}
create /zookeeper/quota/test/zookeeper_limits 'count=1,bytes=100'
create /zookeeper/quota/test/zookeeper_stats 'count=1,bytes=100'
{noformat}

and then you update the limits znode with bogus data, i.e.:

{noformat}
set /zookeeper/quota/test/zookeeper_limits ''
{noformat}

you'll crash the cluster because IllegalArgumentException isn't handled when dealing with quotas znodes:

https://github.com/apache/zookeeper/blob/ZOOKEEPER-823/src/java/main/org/apache/zookeeper/server/DataTree.java#L379
https://github.com/apache/zookeeper/blob/ZOOKEEPER-823/src/java/main/org/apache/zookeeper/server/DataTree.java#L425

We should handle IllegalArgumentException. Optionally, we should also throw BadArgumentsException from PrepRequestProcessor.

Review Board: https://reviews.apache.org/r/25968/
9223372036854775807 No Perforce job exists for this issue. 5 9223372036854775807
1 year, 17 weeks, 1 day ago 0|i1z94v:
ZooKeeper ZOOKEEPER-2018

Zookeper node fails to boot if writes are reordered

Bug Open Major Unresolved Unassigned Samer Al-Kiswany Samer Al-Kiswany 22/Aug/14 14:17   22/Aug/14 21:43   3.4.6       0 2   After studying the steps ZooKeeper takes to update the logs we found the following bug. The bug may manifest in file systems with writeback buffering.

If you run the zookeeper client script (zkCli.sh) with the following commands:
VALUE=”8KB value” # 8KB in size
create /dir1 $VALUE
create /dir1/dir2 $VALUE

the strace generated at the zookeeprer node is:
mkdir(v)
create(v/log)
append(v/log)
trunk(v/log)

fdatasync(v/log)
write(v/log) ……. 1
write(v/log) ……. 2
write(v/log) ……. 3
fdatasync(v/log)

The last four calls are related to the second create of dir2.

If the last write (#3) goes to disk before the second write (#2) and the system crashes before #2 reaches the disk, the zookeeper node will not boot.
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
5 years, 30 weeks, 5 days ago 0|i1z8nb:
ZooKeeper ZOOKEEPER-2017

New tests for reconfig failure cases

Test Resolved Minor Fixed Alexander Shraer Alexander Shraer Alexander Shraer 22/Aug/14 10:10   07/May/15 14:35 24/Aug/14 01:46 3.5.0 3.5.1, 3.6.0 tests   0 5   1) New test file with some reconfig failure cases.
2) Moved testLeaderTimesoutOnNewQuorum from ReconfigTest to the new file
3) Added a check to standaloneDisabledTest.java
9223372036854775807 No Perforce job exists for this issue. 3 9223372036854775807
4 years, 46 weeks ago 0|i1z8bz:
ZooKeeper ZOOKEEPER-2016

Automate client-side rebalancing

Improvement Open Major Unresolved Michael Han Hongchao Deng Hongchao Deng 21/Aug/14 13:23   01/Nov/16 20:12           0 8   ZOOKEEPER-2037 ZOOKEEPER-1355 introduced client-side rebalancing, which is implemented in both the C and Java client libraries. However, it requires the client to detect a configuration change and call updateServerList with the new connection string (see reconfig manual). It may be better if the client just indicates that he is interested in this feature when creating a ZK handle and we'll detect configuration changes and invoke updateServerList for him underneath the hood.

Reviewboard: https://reviews.apache.org/r/25599/
9223372036854775807 No Perforce job exists for this issue. 6 9223372036854775807
3 years, 50 weeks, 6 days ago 0|i1z6yv:
ZooKeeper ZOOKEEPER-2015

I found memory leak in zk client for c++

Bug Resolved Minor Not A Problem Unassigned gao fengpu gao fengpu 21/Aug/14 06:00   12/Nov/15 18:11 25/Aug/14 01:55 3.4.6 3.4.6 c client   0 5   linux/centos 6.3 ==15070== 895,632 bytes in 57,640 blocks are indirectly lost in loss record 370 of 371
==15070== at 0x4C2677B: calloc (vg_replace_malloc.c:593)
==15070== by 0x4C59BB: deserialize_String_vector (zookeeper.jute.c:245)
==15070== by 0x4C5AE7: deserialize_GetChildrenResponse (zookeeper.jute.c:874)
==15070== by 0x4BEE7E: zookeeper_process (zookeeper.c:1906)
==15070== by 0x4BFF8E: do_io (mt_adaptor.c:439)
==15070== by 0x4E36850: start_thread (in /lib64/libpthread-2.12.so)
==15070== by 0x58D367C: clone (in /lib64/libc-2.12.so)
==15070==
==15070== 1,946,648 (1,051,016 direct, 895,632 indirect) bytes in 64,035 blocks are definitely lost in loss record 371 of 371
==15070== at 0x4C2677B: calloc (vg_replace_malloc.c:593)
==15070== by 0x4C59BB: deserialize_String_vector (zookeeper.jute.c:245)
==15070== by 0x4C5AE7: deserialize_GetChildrenResponse (zookeeper.jute.c:874)
==15070== by 0x4BEE7E: zookeeper_process (zookeeper.c:1906)
==15070== by 0x4BFF8E: do_io (mt_adaptor.c:439)
==15070== by 0x4E36850: start_thread (in /lib64/libpthread-2.12.so)
==15070== by 0x58D367C: clone (in /lib64/libc-2.12.so)
9223372036854775807 No Perforce job exists for this issue. 0 9223372036854775807
4 years, 19 weeks ago it's not a bug. the client user muster release the memory manually. 0|i1z6fr:
ZooKeeper ZOOKEEPER-2014

Only admin should be allowed to reconfig a cluster

Bug Closed Blocker Fixed Michael Han Raúl Gutiérrez Segalés Raúl Gutiérrez Segalés 20/Aug/14 22:57   17/May/17 23:44 13/Nov/16 15:10 3.5.0 3.5.3, 3.6.0 server   0 15   ZOOKEEPER-107 introduces reconfiguration support via the reconfig() call. We should, at the very least, ensure that only the Admin can reconfigure a cluster. Perhaps restricting access to /zookeeper/config as well, though this is debatable. Surely one could ensure Admin only access via an ACL, but that would leave everyone who doesn't use ACLs unprotected. We could also force a default ACL to make it a bit more consistent (maybe).

Finally, making reconfig() only available to Admins means they have to run with zookeeper.DigestAuthenticationProvider.superDigest (which I am not sure if everyone does, or how would it work with other authentication providers).

Review board https://reviews.apache.org/r/51546/
9223372036854775807 No Perforce job exists for this issue. 15 9223372036854775807
2 years, 45 weeks, 2 days ago 0|i1z667:
ZooKeeper ZOOKEEPER-2013

typos in zookeeperProgrammers

Bug Resolved Trivial Fixed Tim Chambers Tim Chambers Tim Chambers 19/Aug/14 18:12   30/Aug/14 14:48 20/Aug/14 13:21   3.5.1, 3.6.0 documentation   0 6 60 60 0% I noticed a couple typos. See patch. 0% 0% 60 60 9223372036854775807 No Perforce job exists for this issue. 1 9223372036854775807
5 years, 31 weeks ago 0|i1z2hz:
ZooKeeper ZOOKEEPER-2012

HBase client hangs after client-side OOM

Bug Patch Available Minor Unresolved Qiang Tian Qiang Tian Qiang Tian 15/Aug/14 00:08   24/Aug/14 00:13           0 3   please see http://apache-hbase.679495.n3.nabble.com/HBase-client-hangs-after-client-side-OOM-td4062675.html.

it looks the send thread caught the error successfully, as it is finally running fine.. but the cleanup fail to notify the main thread...so I suspect it is a very small timing hole that the packet is not on the 2 queues at the same time..it looks it could happen in the latest code ClientCnxnSocketNIO#doIO as well..

potential fixes:
1)add timeout during wait
2)try/catch for the possible timing hole:
{code}
if (!p.bb.hasRemaining()) {
sentCount++;
outgoingQueue.removeFirstOccurrence(p);
if (p.requestHeader != null
&& p.requestHeader.getType() != OpCode.ping
&& p.requestHeader.getType() != OpCode.auth) {
synchronized (pendingQueue) {
pendingQueue.add(p);
}
}
}
{code}

thoughts?
thanks.
412271 No Perforce job exists for this issue. 1 412258
5 years, 30 weeks, 4 days ago 0|i1yx9b:
ZooKeeper ZOOKEEPER-2011

move legacy zookeeper jar to dist-maven

Improvement Patch Available Minor Unresolved Qiang Tian Qiang Tian Qiang Tian 14/Aug/14 23:34   21/Aug/14 18:01       build   0 1   Since most users do not care about the source code, it would be better to put the fat jar to disk-maven dir.
also see the README.txt:
{quote}
zookeeper-<version>.jar - legacy jar file which contains all classes
and source files. Prior to version 3.3.0 this
was the only jar file available. It has the
benefit of having the source included (for
debugging purposes) however is also larger as
a result
{quote}
412268 No Perforce job exists for this issue. 2 412255
5 years, 31 weeks, 6 days ago 0|i1yx8n:
ZooKeeper ZOOKEEPER-2010

Add logs when ZooKeeper is back running

Improvement Open Minor Unresolved Unassigned Benjamin Jaton Benjamin Jaton 14/Aug/14 14:27   14/Aug/14 14:27   3.4.6   server   1 3   NIOServerCnxn produces "Zookeeper not running" logs, which is very useful.
It would be also useful to have a way to know when Zookeeper has recovered from it and it running again.
412151 No Perforce job exists for this issue. 0 412140
5 years, 32 weeks ago 0|i1ywjb:
ZooKeeper ZOOKEEPER-2009

zkCli does not execute command passed as arguments

Bug Resolved Minor Duplicate Unassigned Simon Cooper Simon Cooper 14/Aug/14 05:39   16/Aug/16 19:45 16/Aug/16 19:45 3.4.6       0 4   In 3.4.5, zkCli executed commands passed on the command line. This command would create the {{/test}} znode and exit, with a non-zero exit code if the command failed:

{code}
$ ./zkCli.sh create /test null
{code}

This is no longer the case in 3.4.6 - the command is not executed, but zkCli still runs & exits with a zero exit code.

The interim workaround in bash is to use here documents:

{code}
$ ./zkCli.sh <<EOF
create /test null
EOF
{code}
regression 412043 No Perforce job exists for this issue. 0 412032
3 years, 31 weeks, 2 days ago 0|i1yvvr:
ZooKeeper ZOOKEEPER-2008

System test fails due to missing leader election port

Bug Resolved Minor Fixed Kfir Lev-Ari Kfir Lev-Ari Kfir Lev-Ari 12/Aug/14 05:20   30/Aug/14 14:47 14/Aug/14 03:37 3.5.0 3.5.1, 3.6.0 contrib-fatjar   0 4 0 0 0% Leader election and client ports are not initialized when creating a QuorumServer during system tests. 0% 0% 0 0 411455 No Perforce job exists for this issue. 2 411446
5 years, 31 weeks, 6 days ago
Reviewed
0|i1yscf:
ZooKeeper ZOOKEEPER-2007

Update RPM package to be relocatable and contrib packaging bugfix

Bug Resolved Major Won't Fix Eric Yang Eric Yang Eric Yang 08/Aug/14 17:37   03/Mar/16 11:23 03/Mar/16 11:23 3.4.6, 3.5.0       0 1   RPM package init.d startup script is not relocatable, and there is some bugs in contrib directory build structure where property is not passed from main project to contrib, hence some of the contrib projects generate ${dist.dir} directory instead of building in the top level build directory. The usage of BUILD directory is not exactly correct from ZOOKEEPER-1210. RPM build procedure should use BUILDROOT as install destination to properly support RPM 4.6+ while package is building. 410968 No Perforce job exists for this issue. 3 410961
4 years, 3 weeks ago 0|i1ype7:
ZooKeeper ZOOKEEPER-2006

Standalone mode won't take client port from dynamic config

Bug Resolved Major Fixed Hongchao Deng Hongchao Deng Hongchao Deng 07/Aug/14 18:26   30/Aug/14 14:29 19/Aug/14 16:53 3.5.0 3.5.1, 3.6.0 server   0 5   When clientPort is specified in the new format, using "server.x=host:port1:port2;clientPort" in either static or dynamic file and without a "clientPort = xxxx" statement, a standalone mode server doesn't set up client port.

A second problem is that zkServer.sh looks for the client port in both static and dynamic files, but when looking in the static files it only looks for the "clientPort" statement, so if its specified in the new format the port will be missed and commands such as "zkServer.sh status" will not work. This is a problem for standalone mode, but also in distributed mode when the server is still LOOKING (once a leader is established and the server is LEADING/FOLLOWING/OBSERVING, a dynamic file is created and the client port will be found by the script).

Review Board:
https://reviews.apache.org/r/24786/
410735 No Perforce job exists for this issue. 8 410728
5 years, 31 weeks, 1 day ago 0|i1ynz3:
ZooKeeper ZOOKEEPER-2005

Failure to setCurrentEpoch on lead

Bug Open Major Unresolved Unassigned Ioannis Canellos Ioannis Canellos 07/Aug/14 10:10   26/Aug/15 16:46   3.4.6   leaderElection   0 3   We are embedding the zookeeper server in our container and every now and then I see the exception below when running our integration tests suite.

This is something that have never bother us before when using 3.4.5 but we do see in 3.4.6.

When this occurs, the ensemble is not formed.


java.io.IOException: Could not rename temporary file /data/zookeeper/0001/version-2/currentEpoch.tmp to /data/zookeeper/0001/version-2/currentEpoch
at org.apache.zookeeper.common.AtomicFileOutputStream.close(AtomicFileOutputStream.java:82)
at org.apache.zookeeper.server.quorum.QuorumPeer.writeLongToFile(QuorumPeer.java:1202)
at org.apache.zookeeper.server.quorum.QuorumPeer.setCurrentEpoch(QuorumPeer.java:1223)
at org.apache.zookeeper.server.quorum.Leader.lead(Leader.java:395)
410583 No Perforce job exists for this issue. 1 410577
4 years, 30 weeks, 1 day ago 0|i1yn1r:
ZooKeeper ZOOKEEPER-2004

zkCli doesn't output command

Bug Resolved Major Duplicate Unassigned Nelson Nelson 05/Aug/14 03:47   10/Aug/16 15:32 10/Aug/16 15:32 3.4.6   scripts   1 6   linux Hi,

With zookeeper 3.3.6, output is as expected (cf last line which returns the result of ls /
{code}
nelson@nelson-laptop (0) $ ./zookeeper-3.3.6/bin/zkCli.sh -server 127.0.0.1:2181 ls /
Connecting to 127.0.0.1:2181
.... LOGS ....
2014-08-04 16:22:53,032 - INFO [main:Environment@97] - Client environment:user.name=nelson
2014-08-04 16:22:53,032 - INFO [main:Environment@97] - Client environment:user.home=/home/nelson
2014-08-04 16:22:53,033 - INFO [main:Environment@97] - Client environment:user.dir=/home/nelson/git/
2014-08-04 16:22:53,035 - INFO [main:ZooKeeper@379] - Initiating client connection, connectString=127.0.0.1:2181 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@75af8109
2014-08-04 16:22:53,056 - INFO [main-SendThread():ClientCnxn$SendThread@1058] - Opening socket connection to server /127.0.0.1:2181
2014-08-04 16:22:53,158 - INFO [main-SendThread(127.0.0.1:2181):ClientCnxn$SendThread@947] - Socket connection established to127.0.0.1:2181, initiating session
2014-08-04 16:22:53,216 - INFO [main-SendThread(127.0.0.1:2181):ClientCnxn$SendThread@736] - Session establishment complete on server 127.0.0.1:2181, sessionid = 0x147a10f7d02005a, negotiated timeout = 30000

WATCHER::

WatchedEvent state:SyncConnected type:None path:null
[kafka, zookeeper, mesos, marathon, chronos]
nelson@nelson-laptop (0) $
{code}

With zookeeper 3.4.6 no output
{code}
nelson@nelson-laptop (0) $ ./zookeeper-3.4.6/bin/zkCli.sh -server 127.0.0.1:2181 ls /
Connecting to 127.0.0.1:2181
.... LOGS ....
2014-08-04 16:22:56,480 [myid:] - INFO [main:Environment@100] - Client environment:user.name=nelson
2014-08-04 16:22:56,480 [myid:] - INFO [main:Environment@100] - Client environment:user.home=/home/nelson
2014-08-04 16:22:56,480 [myid:] - INFO [main:Environment@100] - Client environment:user.dir=/home/nelson/git/
2014-08-04 16:22:56,481 [myid:] - INFO [main:ZooKeeper@438] - Initiating client connection, connectString=127.0.0.1:2181 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@87d53d8

nelson@nelson-laptop (0) $
{code}
409987 No Perforce job exists for this issue. 0 409981
3 years, 32 weeks, 1 day ago zkcli 0|i1yjev:
ZooKeeper ZOOKEEPER-2003

Missing fsync() on the logs parent directory

Bug Open Major Unresolved Unassigned Samer Al-Kiswany Samer Al-Kiswany 05/Aug/14 03:04   05/Aug/14 13:49   3.4.6       0 4   After studying the steps ZooKeeper takes to update the logs we found the following bug. The bug may not manifest in the current file system implementations, but it violates the POSIX recommendations and may be an issue in some file systems.

Looking at the strace of zookeeper we see the following:
mkdir(v)
create(v/log)
append(v/log)
trunk(v/log)
write(v/log)
fdatasync(v/log)

Although the data is fdatasynced to the log, the parent directory was never fsynced, consequently in case of a crash, the parent directory or the log file may be lost, as the parent directory and file metadata were never persisted on disk.
To be safe, both the log directory, and parent directory of the log directory should be fsynced as well.
409977 No Perforce job exists for this issue. 0 409971
5 years, 33 weeks, 2 days ago 0|i1yjcn:
ZooKeeper ZOOKEEPER-2002

Host

Bug Resolved Major Invalid Unassigned Ruel Balabis Ruel Balabis 03/Aug/14 04:06   03/Aug/14 04:19 03/Aug/14 04:19         0 1 2246400 2246400 0% 0% 0% 2246400 2246400 409623 No Perforce job exists for this issue. 0 409618
5 years, 33 weeks, 4 days ago 0|i1yh7z:
ZooKeeper ZOOKEEPER-2001

Please provide a parse method with input stream as parameter

Improvement Open Major Unresolved Unassigned Suresh Mathew Suresh Mathew 31/Jul/14 18:37   21/Aug/14 18:38   3.4.6       0 2   org.apache.zookeeper.server.quorum.QuorumPeerConfig has the following two methods.
1.public void parse(String path) throws ConfigException
2. public void parseProperties(Properties zkProp)
It would be great if you could please add a wrapper to take an input stream. In the first method, half way through it becomes a file input stream. So I assume its fairly easy to add this wrapper. The reason is most applications will be getting a stream with the classloader help.
409271 No Perforce job exists for this issue. 0 409267
5 years, 31 weeks ago 0|i1yf3b:
ZooKeeper ZOOKEEPER-2000

ZOOKEEPER-2135 Fix ReconfigTest.testPortChange

Sub-task Open Minor Unresolved Alexander Shraer Alexander Shraer Alexander Shraer 31/Jul/14 02:49   05/Feb/20 07:16   3.5.0 3.7.0, 3.5.8 tests   0 6   testPortChange changes all ports and role of the server and thus causes existing clients to disconnect, while this wouldn't happen if only the client port changes. Need to fix it to only change client port and not all the other parameters and make sure that the clients don't disconnect, while new clients shouldn't be able to connect to the old port.
409074 No Perforce job exists for this issue. 2 409070
5 years, 1 week, 5 days ago 0|i1ydvr:
ZooKeeper ZOOKEEPER-1999

Converting CRLF to LF in DynamicConfigBackwardCompatibilityTest

Bug Resolved Major Fixed Hongchao Deng Hongchao Deng Hongchao Deng 30/Jul/14 14:09   01/Aug/14 17:03 01/Aug/14 10:59 3.5.0 3.5.0     0 3   The gitattributes set java files line ending be LF.
The DynamicConfigBackwardCompatibilityTest.java uses CRLF and should be converted to LF.
408924 No Perforce job exists for this issue. 1 408922
5 years, 33 weeks, 6 days ago 0|i1ycz3:
ZooKeeper ZOOKEEPER-1998

C library calls getaddrinfo unconditionally from zookeeper_interest

Bug Open Major Unresolved Raúl Gutiérrez Segalés Raúl Gutiérrez Segalés Raúl Gutiérrez Segalés 29/Jul/14 15:27   14/Dec/19 06:07   3.5.0 3.7.0 c client   1 13 0 3000   (commented this on ZOOKEEPER-338)

I've just noticed that we call getaddrinfo from zookeeper_interest... on every call. So from zookeeper_interest we always call update_addrs:

https://github.com/apache/zookeeper/blob/trunk/src/c/src/zookeeper.c#L2082

which in turns unconditionally calls resolve_hosts:

https://github.com/apache/zookeeper/blob/trunk/src/c/src/zookeeper.c#L787

which does the unconditional calls to getaddrinfo:

https://github.com/apache/zookeeper/blob/trunk/src/c/src/zookeeper.c#L648

We should fix this since it'll make 3.5.0 slower for people relying on DNS. I think this is happened as part of ZOOKEEPER-107 in which the list of servers can be updated.

cc: [~shralex], [~phunt], [~fpj]
100% 100% 3000 0 pull-request-available 408654 No Perforce job exists for this issue. 0 408652
29 weeks ago 0|i1ybc7:
ZooKeeper ZOOKEEPER-1997

server with a single line server list shouldn't be StandaloneEnabled

Bug Resolved Major Duplicate Unassigned Hongchao Deng Hongchao Deng 28/Jul/14 16:22   14/Aug/14 18:48 14/Aug/14 18:48         0 4   A server goes to standalone mode if there is only a single server line in server list description.

In fact, the server line was ignored.

The test [testStandaloneQuorum|https://github.com/apache/zookeeper/blob/fa6d21fa8a8acba812237538e4f7172faf969d37/src/java/test/org/apache/zookeeper/test/StandaloneTest.java#L64] was incorrectly successful before -- the client port was ignored and the server was responding through the jetty port.

When I do a client port check, it failed.

This is caused by the logic in [checkvalidity|https://github.com/apache/zookeeper/blob/4bb76bd22916de8dcfe0c40f649d02d61737e871/src/java/main/org/apache/zookeeper/server/quorum/QuorumPeerConfig.java#L439]:
{code}
if (numMembers > 1 || (!standaloneEnabled && numMembers > 0)) {
...
{code}
This would assume it's standaloneEnabled mode and won't take anything in server list where the client port is defined as introduced in 3.5 dynamic config format.

This is undesired after introducing reconfig because a cluster could set up one server and then add more later.
408373 No Perforce job exists for this issue. 0 408375
5 years, 34 weeks, 2 days ago 0|i1y9of:
ZooKeeper ZOOKEEPER-1996

Incorrect statement in documentation

Bug Resolved Major Duplicate Unassigned Dmitry Sivachenko Dmitry Sivachenko 28/Jul/14 09:17   18/Nov/15 18:27 18/Nov/15 18:27 3.4.6   documentation   1 2   FreeBSD Documentation contains the following warning about FreeBSD:
http://zookeeper.apache.org/doc/r3.4.6/zookeeperAdmin.html#sc_systemReq
-------
FreeBSD is supported as a development and production platform for clients only. Java NIO selector support in the FreeBSD JVM is broken.
-------

I believe it is outdated info from pre-OpenJDK time. With recent OpenJDK-7
I am running Zookeeper in production without any problems and I asked other people who run it on FreeBSD, they also experience no trouble.

I propose to remove this information and list FreeBSD as supported platform unless you know something bad in particular.
408265 No Perforce job exists for this issue. 0 408269
4 years, 18 weeks, 1 day ago 0|i1y90v:
ZooKeeper ZOOKEEPER-1995

ZOOKEEPER-1992 Safely remove client port in old config file on reconfig itself

Sub-task Resolved Major Duplicate Hongchao Deng Hongchao Deng Hongchao Deng 27/Jul/14 22:09   01/Aug/14 01:47 01/Aug/14 01:47 3.5.0 3.5.0     0 1   1. check on reconfig the clientPort field.
2.
408181 No Perforce job exists for this issue. 0 408185
5 years, 34 weeks, 3 days ago 0|i1y8i7:
ZooKeeper ZOOKEEPER-1994

Backup config files.

Improvement Resolved Major Fixed Hongchao Deng Hongchao Deng Hongchao Deng 27/Jul/14 22:08   30/Aug/14 14:29 14/Aug/14 02:19 3.5.0 3.5.1, 3.6.0 server   0 5   We should create a backup file for a static or dynamic configuration file before changing the file.

Since the static file is changed at most twice (once when removing the ensemble definitions, at which point a dynamic file doesn't exist yet, and once when removing clientPort information) its probably fine to back up the static file independently from the dynamic file.

To track backup history:

Option 1: we could have a .bakXX extention for backup where XX is a sequence number.

Option 2: have the configuration version be part of the file name for dynamic configuration files (instead of in the file like now). Such as zoo_replicated1.cfg.dynamic.1000000 then on reconfiguration simply create a new dynamic file (with new version) and update the link in the static file to point to the new dynamic one.

Review place:
https://reviews.apache.org/r/24208/
408180 No Perforce job exists for this issue. 12 408184
5 years, 32 weeks ago
Reviewed
0|i1y8hz:
ZooKeeper ZOOKEEPER-1993

ZOOKEEPER-1992 Keep the client port upon parsing config

Sub-task Resolved Blocker Duplicate Hongchao Deng Hongchao Deng Hongchao Deng 27/Jul/14 22:07   01/Aug/14 01:46 01/Aug/14 01:46 3.5.0 3.5.0     0 3   1. Current implementation ignored and removed "clientPort" on parsing. For the sake of backward compatibility, "clientPort" should be kept and used upon parsing config on fresh boot.
2. When getting clientPort from both the old config and dynamic file, the one in dynamic file is of higher priority.
3. When "dynamicConfigFile" is set in zoo.cfg and not empty, standalone mode will be disabled.

Review board:
https://reviews.apache.org/r/24074/
408179 No Perforce job exists for this issue. 4 408183
5 years, 34 weeks, 1 day ago 0|i1y8hr:
ZooKeeper ZOOKEEPER-1992

backward compatibility of zoo.cfg

Bug Resolved Blocker Fixed Hongchao Deng Hongchao Deng Hongchao Deng 27/Jul/14 22:05   02/Aug/14 15:00 01/Aug/14 01:43 3.5.0 3.5.0     0 4   ZOOKEEPER-1993, ZOOKEEPER-1995 This issue supersedes our discussion in ZOOKEEPER-1989.

To summarize, ZK users can seamlessly upgrade 3.4 to 3.5. But two things will happen:
1. the server list will be separated out as a dynamic file (the original should be backup automatically).
2. Client port is mandatory on reconfig. So when reconfig the server itself (its id), the client port in config file will be removed and replaced by the one in reconfig (written in dynamic file).
408178 No Perforce job exists for this issue. 11 408182
5 years, 33 weeks, 5 days ago
Reviewed
0|i1y8hj:
ZooKeeper ZOOKEEPER-1991

zkServer.sh returns with a zero exit status when a ZooKeeper process is already running

Bug Closed Minor Fixed Biju Nair Reed Wanderman-Milne Reed Wanderman-Milne 25/Jul/14 21:29   21/Jul/16 16:18 02/Mar/16 13:33 3.4.6 3.5.2, 3.6.0 scripts   0 5   If ZooKeeper is started with zkServer.sh, and an error is shown that a ZooKeeper process is already running, the command returns with an exit status of 0, while it should end with a non-zero exit status.

Example:
$ bin/zkServer.sh start
JMX enabled by default
Using config: /home/reed/zookeeper/bin/../conf/zoo.cfg
Starting zookeeper ... already running as process 25063.
$ echo $?
0

This can make it difficult for automated scripts to check if creating a new ZooKeeper process was successful, as it won't catch if a user accidentally started it before.
408069 No Perforce job exists for this issue. 1 408076
4 years, 3 weeks, 1 day ago 0|i1y7tz:
ZooKeeper ZOOKEEPER-1990

suspicious instantiation of java Random instances

Bug Closed Critical Fixed Norbert Kalmár Patrick D. Hunt Patrick D. Hunt 25/Jul/14 19:33   20/May/19 13:51 10/Sep/18 05:45 3.5.0 3.6.0, 3.5.5     0 8 0 2400   It's not clear to me why we are doing this, but it looks very suspicious. Why aren't we just calling "new Random()" in these cases? (even for the tests I don't really see it - typically that would just be for repeatability)

{noformat}
ag "new Random[ \t]*\(" .
src/java/main/org/apache/zookeeper/ClientCnxn.java
817: private Random r = new Random(System.nanoTime());

src/java/main/org/apache/zookeeper/client/StaticHostProvider.java
75: sourceOfRandomness = new Random(System.currentTimeMillis() ^ this.hashCode());
98: sourceOfRandomness = new Random(randomnessSeed);

src/java/main/org/apache/zookeeper/server/quorum/AuthFastLeaderElection.java
420: rand = new Random(java.lang.Thread.currentThread().getId()

src/java/main/org/apache/zookeeper/server/SyncRequestProcessor.java
64: private final Random r = new Random(System.nanoTime());

src/java/main/org/apache/zookeeper/server/ZooKeeperServer.java
537: Random r = new Random(id ^ superSecret);
554: Random r = new Random(sessionId ^ superSecret);

src/java/test/org/apache/zookeeper/server/quorum/WatchLeakTest.java
271: Random r = new Random(SESSION_ID ^ superSecret);

src/java/test/org/apache/zookeeper/server/quorum/CommitProcessorTest.java
151: Random rand = new Random(Thread.currentThread().getId());
258: Random rand = new Random(Thread.currentThread().getId());
288: Random rand = new Random(Thread.currentThread().getId());

src/java/test/org/apache/zookeeper/test/StaticHostProviderTest.java
40: private Random r = new Random(1);

{noformat}
100% 100% 2400 0 pull-request-available 408058 No Perforce job exists for this issue. 0 408066
1 year, 27 weeks, 3 days ago 0|i1y7rr:
ZooKeeper ZOOKEEPER-1989

ZOOKEEPER-1987 backward compatibility of zoo.cfg

Sub-task Resolved Blocker Fixed Hongchao Deng Hongchao Deng Hongchao Deng 24/Jul/14 13:38   01/Aug/14 10:59 01/Aug/14 10:59 3.5.0 3.5.0 tests   0 2   Before 3.5.x, users define zoo.cfg with "clientPort" parameter which is used to identify on which port the server is serving clients.

After upgrading to 3.5.x, the new format:
{noformat}
server.$id=$addr:$port1:$port2[:$role];[$cliAddr:]$cliPort
{noformat}
force users to define all the client ports on the entire ZK ensemble.

The goal of this issue is to preserve backward compatibility upgrading 3.4 to 3.5.
1. when a user defines an old-style config file, it should function the same as the old way -- It should use clientPort variable and shouldn't create a dynamic file.
2. when a user with old-style config file tries to do reconfig relevant jobs, it should stop him and give out a warning.
407582 No Perforce job exists for this issue. 0 407596
5 years, 34 weeks, 6 days ago 0|i1y4wf:
ZooKeeper ZOOKEEPER-1988

ZOOKEEPER-1987 new test patch to verify dynamic reconfig backward compatibility

Sub-task Resolved Major Fixed Alexander Shraer Raúl Gutiérrez Segalés Raúl Gutiérrez Segalés 24/Jul/14 13:26   26/Jul/14 07:25 25/Jul/14 22:16   3.5.0 tests   0 5   407576 No Perforce job exists for this issue. 5 407590
5 years, 34 weeks, 5 days ago 0|i1y4v3:
ZooKeeper ZOOKEEPER-1987

unable to restart 3 node cluster

Bug Resolved Blocker Fixed Alexander Shraer Patrick D. Hunt Patrick D. Hunt 23/Jul/14 20:02   07/Nov/14 12:25 07/Nov/14 12:25 3.5.0 3.5.1 tests   0 6   ZOOKEEPER-1660, ZOOKEEPER-1988, ZOOKEEPER-1989 I tried a fairly simple test, start a three node cluster, bring it down, then restart it. On restart the servers elect the leader and send updates, however the negotiation never completes - the client ports are never bound for example. 407380 No Perforce job exists for this issue. 7 407394
5 years, 35 weeks ago 0|i1y3nr:
ZooKeeper ZOOKEEPER-1986

refactor log trace on touchSession

Improvement Resolved Minor Fixed Hongchao Deng Hongchao Deng Hongchao Deng 23/Jul/14 17:22   25/Jul/14 07:25 24/Jul/14 18:59   3.5.0     0 2   The previous log trace has a minor mistake after applying ZOOKEEPER-1982:
It will show "invalidsession" and "closingsession". There should be a whitespace in between.

I might wanna further refactor the log wrapper function by using MessageFormat, which would be cleaner.
407342 No Perforce job exists for this issue. 1 407356
5 years, 34 weeks, 6 days ago
Reviewed
0|i1y3fb:
ZooKeeper ZOOKEEPER-1985

Memory leak in C client

Bug Open Major Unresolved desmondhe desmondhe desmondhe 22/Jul/14 22:22   05/Feb/20 07:16   3.4.6 3.7.0, 3.5.8 c client   0 4   in the file zookeeper.c, most function call of "close_buffer_oarchive(&oa, 0)" shoud been instead by
close_buffer_oarchive(&oa, rc < 0 ? 1 : 0);
407142 No Perforce job exists for this issue. 1 407158
1 year, 17 weeks ago 0|i1y27b:
ZooKeeper ZOOKEEPER-1984

testLeaderTimesoutOnNewQuorum is a flakey test

Bug Resolved Major Fixed Alexander Shraer Patrick D. Hunt Patrick D. Hunt 22/Jul/14 17:37   24/Jul/14 07:22 23/Jul/14 12:37 3.5.0 3.5.0 tests   0 3   I'm seeing intermittent failures in testLeaderTimesoutOnNewQuorum

It's failing both on jdk6 and jdk7. (this is my personal
jenkins, I haven't see any other failures than this during the past
few days).

{noformat}
junit.framework.AssertionFailedError
at org.apache.zookeeper.test.ReconfigTest.testServerHasConfig(ReconfigTest.java:127)
at org.apache.zookeeper.test.ReconfigTest.testLeaderTimesoutOnNewQuorum(ReconfigTest.java:450)
at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:52)
{noformat}
407056 No Perforce job exists for this issue. 1 407073
5 years, 35 weeks ago
Reviewed
0|i1y1of:
ZooKeeper ZOOKEEPER-1983

Append to zookeeper.out (not overwrite) to support logrotation

Bug In Progress Major Unresolved Shyamal Prasad Shyamal Prasad Shyamal Prasad 22/Jul/14 17:18   05/Feb/20 07:11   3.3.5, 3.3.6, 3.4.6 3.7.0, 3.5.8 server   1 4   CentOS 5.x (and probably any Linux distribution for that matter) Currently zkServer.sh will redirect output to zookeeper.out using a simple shell redirect.

When logrotate (and similar tools) are used to rotate the zookeeper.out file with the 'copytruncate' semantics (copy the file, truncate it to zero bytes) the next write results in a sparse file with the write at the offset of the last file. Effectively the log file is now full a null bytes and it is hard to read/use the file (and the rotated copies).

Even worse, the result is zookeeper.out file only gets "larger" (though sparse) and after a while on a chatty system it takes significant CPU resources to compress the file (which is all nulls!)

The simple fix is to append to the file (>>) instead of a simple redirection (>)

This issue was found in a 3.3.5 production system, however code in trunk has the same issue.
407047 No Perforce job exists for this issue. 2 407065
4 years, 48 weeks, 2 days ago 0|i1y1mn:
ZooKeeper ZOOKEEPER-1982

Refactor (touch|add)Session in SessionTrackerImpl.java

Improvement Resolved Minor Fixed Hongchao Deng Hongchao Deng Hongchao Deng 22/Jul/14 16:07   24/Jul/14 07:22 23/Jul/14 14:17   3.5.0     0 3   This JIRA extends the idea of ZOOKEEPER-1978.

Besides refactoring get-put operations of concurrentMap in addSession method, addSession also calls touchSession which repeatedly checks if session existed. So it would be nice for refactor. Refactoring the second issue is relevant to ZOOKEEPER-1978. So I create a this JIRA to fix both.
407029 No Perforce job exists for this issue. 3 407047
5 years, 35 weeks ago
Reviewed
0|i1y1in:
ZooKeeper ZOOKEEPER-1981

ZOOKEEPER-1970 Fix Dodgy Code Warnings identified by findbugs 2.0.3

Sub-task Resolved Minor Fixed Hongchao Deng Hongchao Deng Hongchao Deng 21/Jul/14 12:45   22/Jul/14 07:31 22/Jul/14 00:51   3.5.0     0 3   There are two cases:
1. a duplicate check of null bytes.
2. a lot switch statement without default case.

For the default case, I suggest
1. throwing an exception as a way to break the program. It's highly unexpected.
2. or LOG.warn it.

I am doing the second right now to keep the original.
406694 No Perforce job exists for this issue. 1 406714
5 years, 35 weeks, 2 days ago
Reviewed
0|i1xzh3:
ZooKeeper ZOOKEEPER-1980

how to draw the figure"ZooKeeper Throughput as the Read-Write Ratio Varies" ?

Test Resolved Major Done Unassigned xinyanzhang xinyanzhang 21/Jul/14 03:31   28/Jul/14 12:57 28/Jul/14 12:57 3.4.6 3.4.6 tests   0 2 972000 972000 0% Each server has one Xeon dual-core 2.4GHz processor, 4GB of RAM, gigabit Ethernet, and two SATA drives. The Linux Kernel version is 2.6.18-164.11.1.el5_lustre.1.8.3, the operating system is Red Hat Enterprise Linux Server release 5.4 (Tikanga), and the java version is 1.7.0_51. I want to know how to draw the figure "ZooKeeper Through as the Read-Write Ratio Varies" and " Reliability in the Presence of Errors" .
is it produced by the benchmarking tools provide by the Computer Science department of Brown University?
0% 0% 972000 972000 406561 No Perforce job exists for this issue. 0 406581
5 years, 34 weeks, 3 days ago 0|i1xynr:
ZooKeeper ZOOKEEPER-1979

ZOOKEEPER-1970 Fix Performance Warnings found by Findbugs 2.0.3

Sub-task Resolved Minor Fixed Hongchao Deng Hongchao Deng Hongchao Deng 20/Jul/14 18:30   22/Jul/14 07:31 22/Jul/14 00:46   3.5.0     0 3   findbugs complains that
{code}
new Integer(cnxToValue);
{code}
should be changed to
{code}
Integer.parseInt(cnxToValue);
{code}
406526 No Perforce job exists for this issue. 1 406546
5 years, 35 weeks, 2 days ago
Reviewed
0|i1xyfz:
ZooKeeper ZOOKEEPER-1978

ZOOKEEPER-1970 Fix Multithreaded correctness Warnings

Sub-task Resolved Minor Duplicate Hongchao Deng Hongchao Deng Hongchao Deng 20/Jul/14 18:28   23/Jul/14 14:18 23/Jul/14 14:18   3.5.0     0 3   findbugs is complaining
{code}
if (sessionsById.get(id) == null) {
SessionImpl s = new SessionImpl(id, sessionTimeout);
sessionsById.put(id, s);
}
{code}
is not atomic for the gap between get() and put().

I suggest using putIfAbsent() instead.
406525 No Perforce job exists for this issue. 1 406545
5 years, 35 weeks, 1 day ago 0|i1xyfr:
ZooKeeper ZOOKEEPER-1977

Calibrate initLimit dynamically

Wish Open Major Unresolved Unassigned Flavio Paiva Junqueira Flavio Paiva Junqueira 20/Jul/14 14:32   11/Aug/14 13:33           2 7   We have seen a number of times users failing to get an ensemble up because the snapshot transfer times out. We should be able to do better than this and calibrate initLimit dynamically. I was thinking concretely that we could have servers increasing the initLimit value (e.g., doubling or increments of 1) upon socket timeouts. The tricky part here is that we need both ends of the communication to increase it. 406515 No Perforce job exists for this issue. 0 406535
5 years, 32 weeks, 3 days ago 0|i1xydj:
ZooKeeper ZOOKEEPER-1976

address internationalization issues identified by findbugs 2.0.3

Bug Open Major Unresolved Unassigned Patrick D. Hunt Patrick D. Hunt 19/Jul/14 23:01   19/Jun/17 06:33   3.5.0, 3.4.11   server   0 2   Findbugs 2.0.3 found a number of internationalization issues with the code, we ignored these for the time being in ZOOKEEPER-1975, however we should address them one by one eventually. 406495 No Perforce job exists for this issue. 0 406515
2 years, 39 weeks, 4 days ago 0|i1xy93:
ZooKeeper ZOOKEEPER-1975

ZOOKEEPER-1970 Turn off "internationalization warnings" in findbugs exclude file

Sub-task Resolved Major Fixed Patrick D. Hunt Patrick D. Hunt Patrick D. Hunt 19/Jul/14 22:53   21/Jul/14 14:57 21/Jul/14 14:01 3.5.0 3.5.0 server   0 3   A more recent version of findbugs (2.0.3) added some checks - one is for internationalization issues. We should fix these, but for the time being we'll ignore them. (I'll also create a separate jira to address this in future) 406494 No Perforce job exists for this issue. 2 406514
5 years, 35 weeks, 3 days ago
Reviewed
0|i1xy8v:
ZooKeeper ZOOKEEPER-1974

winvs2008 jenkins job failing with "unresolved external symbol"

Bug Resolved Blocker Fixed Flavio Paiva Junqueira Patrick D. Hunt Patrick D. Hunt 19/Jul/14 12:57   25/Jul/14 13:39 25/Jul/14 12:24 3.5.0 3.5.0 c client   0 4   The winvs2008 build is failing with

bq. unresolved external symbol __imp__ZOO_READONLY_STATE

see: https://builds.apache.org/view/S-Z/view/ZooKeeper/job/ZooKeeper-trunk-WinVS2008/1445/console
406468 No Perforce job exists for this issue. 1 406488
5 years, 34 weeks, 6 days ago
Reviewed
0|i1xy33:
ZooKeeper ZOOKEEPER-1973

Jetty Server changes broke ibm6 support

Bug Resolved Major Fixed Bill Havanki Patrick D. Hunt Patrick D. Hunt 19/Jul/14 12:50   21/Jul/14 14:57 21/Jul/14 13:59 3.5.0 3.5.0 server   0 4   The recent Jetty Server additions ZOOKEEPER-1346 have broken ibm6 support, can someone take a look? (we've had issues like this in the past - typically due to using specialized/sun classes that don't exist in that jdk)

https://builds.apache.org/job/ZooKeeper-trunk-ibm6/556/
406467 No Perforce job exists for this issue. 1 406487
5 years, 35 weeks, 3 days ago
Reviewed
0|i1xy2v:
ZooKeeper ZOOKEEPER-1972

ZOOKEEPER-1970 Fix invalid volatile long/int increment (++)

Sub-task Resolved Major Fixed Hongchao Deng Hongchao Deng Hongchao Deng 18/Jul/14 19:44   20/Jul/14 07:24 19/Jul/14 22:51 3.5.0 3.5.0     0 3   The findbugs complains about incrementing a volatile variable in [AuthFastLeaderElection|https://github.com/apache/zookeeper/blob/087abf83684167ae56864fe4c3be0079fa653266/src/java/main/org/apache/zookeeper/server/quorum/AuthFastLeaderElection.java#L737] and [FastLeaderElection|https://github.com/apache/zookeeper/blob/087abf83684167ae56864fe4c3be0079fa653266/src/java/main/org/apache/zookeeper/server/quorum/FastLeaderElection.java]:
{code}
volatile long logicalclock; /* Election instance */
...
logicalclock++;
{code}

Actually this is a bug. It should use AtomicLong here instead of volatile.

Leader.java and [QuorumPeer.java|https://github.com/apache/zookeeper/blob/087abf83684167ae56864fe4c3be0079fa653266/src/java/main/org/apache/zookeeper/server/quorum/QuorumPeer.java#L428] and LearnerHandler.java:
{code}
volatile int tick;
{code}

I don't think it needs volatile here. The tick is incremented only in [Leader.java|https://github.com/apache/zookeeper/blob/087abf83684167ae56864fe4c3be0079fa653266/src/java/main/org/apache/zookeeper/server/quorum/Leader.java#L590]:
{code}
synchronized (this) {
...
if (!tickSkip) {
self.tick++;
}
}
{code}
and it's protected by the synchronized statement. I just remove volatile keyword.
406410 No Perforce job exists for this issue. 4 406430
5 years, 35 weeks, 4 days ago Awesome! thanks Hongchao! Committed to trunk
Reviewed
0|i1xxq7:
ZooKeeper ZOOKEEPER-1971

Make JMX remote monitoring port configurable

Improvement Patch Available Major Unresolved Mohammad Arshad Biju Nair Biju Nair 18/Jul/14 17:33   05/Feb/20 07:11   3.5.0 3.7.0, 3.5.8 server   0 6   All This is a follow-up item from ZOOKEEPER-1948. 406385 No Perforce job exists for this issue. 6 406405
1 year, 45 weeks ago 0|i1xxkn:
ZooKeeper ZOOKEEPER-1970

Fix Findbugs Warnings

Improvement Resolved Major Implemented Hongchao Deng Hongchao Deng Hongchao Deng 18/Jul/14 13:46   23/Jul/14 14:19 23/Jul/14 14:19   3.5.0     0 3   ZOOKEEPER-1972, ZOOKEEPER-1975, ZOOKEEPER-1978, ZOOKEEPER-1979, ZOOKEEPER-1981 The findbugs complained a lot of warnings after upgrade:
https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/2191//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html#Warnings_I18N

It would be great to get those warnings settled before 3.5.0 release.

My proposal is:
1. Ignore those "Internationalization Warnings" which is related to encoding. And create a new JIRA to fix encoding later.
2. fix warnnings of:
* Multithreaded correctness Warnings
* Performance Warnings
* Dodgy code Warnings
406354 No Perforce job exists for this issue. 0 406375
5 years, 35 weeks, 1 day ago 0|i1xxdz:
ZooKeeper ZOOKEEPER-1969

Fix Port Already In Use for JettyAdminServerTest

Bug Resolved Major Fixed Hongchao Deng Hongchao Deng Hongchao Deng 18/Jul/14 12:47   18/Jul/14 13:56 18/Jul/14 13:05   3.5.0     0 4   The test is failiing:
{code}
failed SocketConnector@0.0.0.0:8080: java.net.BindException: Address already in use
{code}

I tried to assign a unique port in test and Jenkins build comes normal again.
406340 No Perforce job exists for this issue. 1 406361
5 years, 35 weeks, 6 days ago
Reviewed
0|i1xxb3:
ZooKeeper ZOOKEEPER-1968

Make Jetty dependencies optional in ivy.xml

Improvement Resolved Major Fixed Bill Havanki Patrick D. Hunt Patrick D. Hunt 17/Jul/14 20:24   19/Jul/14 07:24 18/Jul/14 13:29 3.5.0 3.5.0 server   0 3   Should we make the jetty/jackson libraries optional in ivy.xml given that Jetty Server is itself optional (an uses reflection to boot). 406197 No Perforce job exists for this issue. 1 406218
5 years, 35 weeks, 5 days ago
Reviewed
0|i1xwfj:
ZooKeeper ZOOKEEPER-1967

Eliminate the temp dynamic config file, find last proposed config in transaction log.

Improvement Open Major Unresolved Unassigned Alexander Shraer Alexander Shraer 16/Jul/14 16:46   18/Jul/14 13:56       quorum, server   0 3   The .next temporary config file is created when a server acks a reconfig proposal.
During reconfig commit this file becomes the permanent dynamic config file.
This temp file is read (if exists) during server boot to determine whether there is a reconfig potentially in progress.

This info is also available in the transaction log, since reconfig is a transaction. Initially I chose not to take this information from the transaction log, mainly for simplicity, since I believed that we need the last proposed reconfig info before we're processing the transaction log (for example, if we'd like to contact new config servers during FLE - this is discussed in ZOOKEEPER-1807).

It would be useful to revisit this issue and check whether we could eliminate the temporary .next dynamic config file, finding the last proposed reconfig in the the transaction log.

Note that a bulk of the work here will be modifying ReconfigRecoveryTest, which uses .next files to start a server in a state where it thinks it crashed in a middle of a reconfig.
405878 No Perforce job exists for this issue. 0 405898
5 years, 36 weeks, 1 day ago 0|i1xuhj:
ZooKeeper ZOOKEEPER-1966

VS and line breaks

Bug Resolved Major Fixed Orion Hodson Orion Hodson Orion Hodson 15/Jul/14 09:52   17/Jul/14 13:23 16/Jul/14 12:01 3.5.0 3.5.0 c client   0 5   Windows, Visual Studio 2013 Build The patch attached to https://issues.apache.org/jira/browse/ZOOKEEPER-1953 has caused problems for git users when committed from SVN.

The attached patch simply changes the line endings of the offending files from CRLF to LF in the hope that when they are committed to SVN, the LF line endings end up as the canonical representation in the Apache ZooKeeper git repo.

An interpretation of what's happened here is that svn has stored the CRLF line endings and these have been pushed into git by git-svn as described here: http://blog.subgit.com/line-endings-handling-in-svn-git-and-subgit/. Git clients are then confused as the text files have an unexpected representation in the repo.

Experimentally VS is indifferent to line endings – ran dos2unix on the vcxproj and sln files and VS opened and closed the files without modifying them. This page seems to advertise the indifference to line endings and discusses selecting custom options: http://msdn.microsoft.com/en-us/library/dd409797.aspx.

405444 No Perforce job exists for this issue. 2 405469
5 years, 36 weeks ago 0|i1xruf:
ZooKeeper ZOOKEEPER-1965

Install could not be done on powerpc Error: Unrecognized opcode: `lock'

Bug Open Major Unresolved Unassigned Sonny Yang Sonny Yang 14/Jul/14 22:36   14/Jul/14 22:36   3.4.5   c client   0 1   powerpc64 Linux version 2.6.32-220.el6.ppc64 (mockbuild@ppc-004.build.bos.redhat.com) (gcc version 4.4.5 20110214 (Red Hat 4.4.5-6) (GCC) ) #1 SMP Wed Nov 9 08:02:37 EST 2011 after (ant compile_jute) and(cd src/c/ ./configure), make can not be done!
It contains error:
/tmp/ccyJ6new.s: Assembler messages:
/tmp/ccyJ6new.s:67: Error: Unrecognized opcode: `lock'
/tmp/ccyJ6new.s:102: Error: Unrecognized opcode: `lock'
/tmp/ccyJ6new.s:431: Error: Unrecognized opcode: `lock'
/tmp/ccyJ6new.s:464: Error: Unrecognized opcode: `lock'
make[1]: *** [libzkmt_la-mt_adaptor.lo] Error 1
make[1]: Leaving directory `/gpfs/ibmu/sjtupower/rawdep/zookeeper-3.4.5/src/c'
make: *** [all] Error 2

I don't know how to fix it
405341 No Perforce job exists for this issue. 0 405368
5 years, 36 weeks, 2 days ago 0|i1xr8n:
ZooKeeper ZOOKEEPER-1964

Fix Flaky Test in ReconfigTest.java

Bug Resolved Minor Fixed Hongchao Deng Hongchao Deng Hongchao Deng 11/Jul/14 19:20   15/Jul/14 07:20 14/Jul/14 13:46   3.5.0     0 5   There is flaky tests in ReconfigTest showing something like:
{code}
junit.framework.AssertionFailedError: Mismatches ElectionAddress! expected:<[127.0.0.1]:12369> but was:<[localhost]:12369>
at org.apache.zookeeper.test.ReconfigTest.assertRemotePeerMXBeanAttributes(ReconfigTest.java:967)
at org.apache.zookeeper.test.ReconfigTest.testJMXBeanAfterRemoveAddOne(ReconfigTest.java:809)
at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:52)
{code}

Basically, the problem is that there might be inconsistency between numerical ip and literal ip. Converting both sides to one (numerical IP) will fix it.
405009 No Perforce job exists for this issue. 2 405045
5 years, 36 weeks, 2 days ago 0|i1xp9z:
ZooKeeper ZOOKEEPER-1963

Make JDK 7 the minimum requirement for Zookeeper

Improvement Resolved Major Fixed Hongchao Deng Edward Carter Edward Carter 10/Jul/14 13:20   18/Dec/14 05:31 17/Dec/14 17:02 3.5.0 3.5.1, 3.6.0 build   0 10   JDK 6 stopped receiving public updates in early 2013:

http://www.oracle.com/technetwork/java/eol-135779.html

I propose making JDK 7 the minimum for Zookeeper going forward. One patch that I've personally submitted already would have been a good fit for Java 7's try-with-resources statement, and another pending patch fails to build on versions of Java prior to 7 because a unit test in it uses InetAddress.getLoopbackAddress(), which would be awkward to replace. I'm sure there are many other examples.
404737 No Perforce job exists for this issue. 2 404775
5 years, 14 weeks ago 0|i1xnm7:
ZooKeeper ZOOKEEPER-1962

Add a CLI command to recursively list a znode and children

New Feature Closed Minor Fixed Gautam Gopalakrishnan Gautam Gopalakrishnan Gautam Gopalakrishnan 10/Jul/14 07:29   17/May/17 23:43 08/Sep/16 16:53 3.4.6 3.5.3, 3.6.0 java client   2 17 86400 86400 0% When troubleshooting applications where znodes can be multiple levels deep (eg. HBase replication), it is handy to see all child znodes recursively rather than run an ls for each node manually.

So I propose adding an option to the "ls" command (-r) which will list all child nodes under a given znode.
0% 0% 86400 86400 404665 No Perforce job exists for this issue. 7 404703
3 years, 28 weeks ago
Reviewed
0|i1xn6f:
ZooKeeper ZOOKEEPER-1961

NPE in ZooKeeperServerMain.shutdown()

Improvement Open Major Unresolved Unassigned Steve Loughran Steve Loughran 08/Jul/14 11:56   09/Jul/14 09:13   3.4.6   server   0 2   while trying to stop a server that appears not to have started properly, the {{shutdown()}} method triggered an NPE 404219 No Perforce job exists for this issue. 0 404259
5 years, 37 weeks, 1 day ago 0|i1xkg7:
ZooKeeper ZOOKEEPER-1960

Improve logs in PurgeTxnLog.java to give more details

Improvement Open Major Unresolved Unassigned nijel nijel 08/Jul/14 10:29   08/Jul/14 10:29           0 0   Improve logs in PurgeTxnLog.java to give more details
Suggest to add logs in following scenarios
1. If no file to purge (deletion list is empty) - INFO
2. Add logs for debug purpose (txnLog, dataDir and snapDir, passed arguements)
log 404198 No Perforce job exists for this issue. 0 404238
5 years, 37 weeks, 2 days ago 0|i1xkbr:
ZooKeeper ZOOKEEPER-1959

Move zkEnv files to conf folder, since it is expected to be configured by user

Improvement Open Major Unresolved Unassigned nijel nijel 08/Jul/14 10:23   08/Jul/14 10:23           0 1   Move zkEnv files to conf folder
This is the common pattern followed across other hadoop components
script 404197 No Perforce job exists for this issue. 0 404237
5 years, 37 weeks, 2 days ago 0|i1xkbj:
ZooKeeper ZOOKEEPER-1958

Client does not detect rejected connections

Bug Open Minor Unresolved Unassigned Erik Anderson Erik Anderson 07/Jul/14 20:32   08/Jul/14 02:33   3.4.6   c client   0 1   Windows 7 64bit When attempting to connect to a zookeeper server that is not currently running, the connection will return the "connection refused" message through the timeout logic. This is because Windows is returning the error code through select->error rather than select->write (which is what the logic is apparently expecting)

Patch is pending
404053 No Perforce job exists for this issue. 2 404094
5 years, 37 weeks, 2 days ago 0|i1xjg7:
ZooKeeper ZOOKEEPER-1957

ReconfigTest can fail when checking remote peer MXBean ElectionAddress

Test Open Minor Unresolved Unassigned Bill Havanki Bill Havanki 07/Jul/14 14:33   07/Jul/14 19:18   3.4.6   jmx   0 1   CThis happens most of the time when I run the {{ReconfigTest}} unit test on Mac OS X Mavericks, using Java 6 or 7. I get failures like this:

{noformat}
Mismatches ElectionAddress! expected:<[127.0.0.1]:12369> but was:<[localhost]:12369>
junit.framework.AssertionFailedError: Mismatches ElectionAddress! expected:<[127.0.0.1]:12369> but was:<[localhost]:12369>
at org.apache.zookeeper.test.ReconfigTest.assertRemotePeerMXBeanAttributes(ReconfigTest.java:967)
at org.apache.zookeeper.test.ReconfigTest.testJMXBeanAfterRoleChange(ReconfigTest.java:887)
at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:52)
{noformat}

It has to do with {{RemotePeerBean.getElectionAddress()}}, which has an InetAddress of "localhost/127.0.0.1" and picks "localhost" over the IP address. I'm not sure if the IP address should have been picked, or if the test should allow "localhost".
testing 403967 No Perforce job exists for this issue. 0 404009
5 years, 37 weeks, 3 days ago 0|i1xixj:
ZooKeeper ZOOKEEPER-1956

Support Cleanup script in windows

Bug Open Minor Unresolved Unassigned nijel nijel 07/Jul/14 06:04   02/Mar/16 20:51   3.4.6, 3.5.0       1 2   The script zkCleanup.sh support cleaning the zk data in linux system.
The same function needs to be supported in windows also
403868 No Perforce job exists for this issue. 0 403911
4 years, 3 weeks ago Supporting cleaning snapshots log history via Windows script. 0|i1xibr:
ZooKeeper ZOOKEEPER-1955

EOFException on Reading Snapshot

Bug Open Major Unresolved Unassigned Aaron Zimmerman Aaron Zimmerman 04/Jul/14 18:30   30/Nov/15 10:13   3.4.5       2 7   We have a 5 node zookeeper cluster that has been operating normally for several months. Starting a few days ago, the entire cluster crashes a few times per day, all nodes at the exact same time. We can't track down the exact issue, but deleting the snapshots and logs and restarting allows the cluster to come back up.

We are running exhibitor to monitor the cluster.

It appears that something bad gets into the logs, causing an EOFException and this cascades through the entire cluster:



2014-07-04 12:55:26,328 [myid:1] - WARN [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:Follower@89] - Exception when following the leader
java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:375)
at org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63)
at org.apache.zookeeper.server.quorum.QuorumPacket.deserialize(QuorumPacket.java:83)
at org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:108)
at org.apache.zookeeper.server.quorum.Learner.readPacket(Learner.java:152)
at org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:85)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:740)
2014-07-04 12:55:26,328 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:Follower@166] - shutdown called
java.lang.Exception: shutdown Follower
at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:166)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:744)


Then the server dies, exhibitor tries to restart each node, and they all get stuck trying to replay the bad transaction, logging things like:


2014-07-04 12:58:52,734 [myid:1] - INFO [main:FileSnap@83] - Reading snapshot /var/lib/zookeeper/version-2/snapshot.300011fc0
2014-07-04 12:58:52,896 [myid:1] - DEBUG [main:FileTxnLog$FileTxnIterator@575] - Created new input stream /var/lib/zookeeper/version-2/log.300000021
2014-07-04 12:58:52,915 [myid:1] - DEBUG [main:FileTxnLog$FileTxnIterator@578] - Created new input archive /var/lib/zookeeper/version-2/log.300000021
2014-07-04 12:59:25,870 [myid:1] - DEBUG [main:FileTxnLog$FileTxnIterator@618] - EOF excepton java.io.EOFException: Failed to read /var/lib/zookeeper/version-2/log.300000021
2014-07-04 12:59:25,871 [myid:1] - DEBUG [main:FileTxnLog$FileTxnIterator@575] - Created new input stream /var/lib/zookeeper/version-2/log.300011fc2
2014-07-04 12:59:25,872 [myid:1] - DEBUG [main:FileTxnLog$FileTxnIterator@578] - Created new input archive /var/lib/zookeeper/version-2/log.300011fc2
2014-07-04 12:59:48,722 [myid:1] - DEBUG [main:FileTxnLog$FileTxnIterator@618] - EOF excepton java.io.EOFException: Failed to read /var/lib/zookeeper/version-2/log.300011fc2

And the cluster is dead. The only way we have found to recover is to delete all of the data and restart.

[~fournc] Appreciate any assistance you can offer.
403690 No Perforce job exists for this issue. 1 403733
4 years, 16 weeks, 3 days ago 0|i1xh8f:
ZooKeeper ZOOKEEPER-1954

StaticHostProvider loses IPv6 scope ID when resolving server addresses

Bug In Progress Minor Unresolved Unassigned Bill Havanki Bill Havanki 03/Jul/14 16:45   18/Jun/15 08:58   3.4.6   java client   2 3   I have been getting constant failures of the {{ClientPortBindTest}} unit test (see ZOOKEEPER-1256) on my Macbook. I traced the problem to loss of the IPv6 scope ID on the address chosen for the loopback address in the unit test.

The address chosen is: fe80:0:0:0:0:0:0:1%1. The scope ID here is 1, after the percent sign.

The scope ID is lost in the {{resolveAndShuffle()}} method of {{StaticHostProvider}}. The method uses {{InetAddress.getByAddress()}} which apparently does not preserve the scope ID in the host string it is passed. {{Inet6Address.getByAddress()}} can, although you have to parse the scope ID out of the host string yourself and pass it as its own parameter.
403535 No Perforce job exists for this issue. 1 403579
Patch
4 years, 40 weeks ago 0|i1xgaf:
ZooKeeper ZOOKEEPER-1953

Add solution and project files to enable build with current Visual Studio editions (VS 2012/2013) - 32-bit and 64-bit.

Improvement Resolved Major Fixed Orion Hodson Patrick D. Hunt Patrick D. Hunt 02/Jul/14 23:49   15/Jul/14 10:09 11/Jul/14 11:52 3.5.0 3.5.0 c client   0 6   Add solution and project files to enable build with current Visual Studio editions (VS 2012/2013) - 32-bit and 64-bit. 403353 No Perforce job exists for this issue. 2 403404
5 years, 36 weeks, 2 days ago 0|i1xf7r:
ZooKeeper ZOOKEEPER-1952

Default log directory and file name can be changed

Bug Resolved Minor Fixed nijel nijel nijel 02/Jul/14 08:26   02/Mar/15 06:14 01/Mar/15 11:50 3.4.6 3.5.1, 3.6.0     0 7   The log folder and log file name is configurable now.
The default log folder is "." in distribution. So the log file (zookeeper.out) will be placed in bin folder

Can this be changed to <zk_home>/logs/zookeeperserver-<hostname>.log ?
403140 No Perforce job exists for this issue. 5 403198
5 years, 3 weeks, 3 days ago 0|i1xdy7:
ZooKeeper ZOOKEEPER-1951

Clean up raw type compile warnings under Java 7

Improvement In Progress Trivial Unresolved Unassigned Bill Havanki Bill Havanki 01/Jul/14 11:57   18/Jun/15 08:58   3.4.6       0 1   Compiling with Java 1.7.0_60 under Mac OS X 10.9.3, I get five warnings about raw types being found. (Compiler warnings are attached.) These can probably be cleaned up pretty easily.

The warnings were observed in ZOOKEEPER-1477.
generics, java7 402940 No Perforce job exists for this issue. 2 403000
5 years, 38 weeks, 2 days ago 0|i1xcpj:
ZooKeeper ZOOKEEPER-1950

configBackwardCompatibilityMode breaks compatibility

Bug Resolved Major Duplicate Unassigned Hongchao Deng Hongchao Deng 30/Jun/14 17:43   24/Jul/14 12:06 24/Jul/14 11:20 3.5.0 3.5.0     0 1   The current implementation divide information of servers of legacy config into two separate dynamic config files. There is a problem.

When we set "clientPort" variable in config file, it gets automatically erased and later on there is no information about "clientPort" in either the old or new (.dynamic) config file.

It becomes a serious problem when users of *3.4* jump to *3.5* directly without changing their config: when a server crashes and restarts, there is no client port serving.

For example,
a legacy config might look like:

```zoo.cfg
dataDir=/root/zookeeper/groupconfig/conf1/data
syncLimit=5
initLimit=10
tickTime=2000
clientPort=2181
server.1=127.0.0.1:2222:2223
server.2=127.0.0.1:3333:3334
server.3=127.0.0.1:4444:4445
```

After dynamic reconfig, it might look like

```zoo.cfg
dataDir=/root/zookeeper/groupconfig/conf1/data
syncLimit=5
tickTime=2000
initLimit=10
dynamicConfigFile=./zoo.cfg.dynamic
```

and
```zoo.cfg.dynamic
server.1=127.0.0.1:2222:2223:participant
server.2=127.0.0.1:3333:3334:participant
server.3=127.0.0.1:4444:4445:participant
version=e00000000
```

This could be successfully started at first time. But when server restarts from crash, it never serve client port again.
402747 No Perforce job exists for this issue. 0 402814
5 years, 35 weeks ago 0|i1xbkv:
ZooKeeper ZOOKEEPER-1949

recipes jar not included in the distribution package

Bug Resolved Major Fixed Rakesh Radhakrishnan Rakesh Radhakrishnan Rakesh Radhakrishnan 27/Jun/14 04:01   11/Feb/15 06:16 11/Feb/15 02:02   3.4.7, 3.5.1, 3.6.0 recipes   0 5   Following recipe jars doesn't exists in the distribution "zookeeper-3.4.6.tar.gz"

recipes/election/zookeeper-3.4.6-recipes-election.jar
recipes/lock/zookeeper-3.4.6-recipes-lock.jar
recipes/queue/zookeeper-3.4.6-recipes-queue.jar
build 402262 No Perforce job exists for this issue. 1 402326
5 years, 6 weeks, 1 day ago 0|i1x8kv:
ZooKeeper ZOOKEEPER-1948

Enable JMX remote monitoring

Improvement Resolved Major Fixed Biju Nair Biju Nair Biju Nair 26/Jun/14 16:21   26/Dec/14 19:08 29/Sep/14 13:18 3.4.6 3.4.7, 3.5.1, 3.6.0 server   0 6   All The zooker server start up script includes the option to enable jmx monitoring but only locally. Can we update the script so that remote monitoring can also be enabled which will help in data collection and monitoring through a centralized monitoring tool. 402140 No Perforce job exists for this issue. 4 402206
5 years, 12 weeks, 6 days ago Changes in zkServer.sh to support JMX remote monitoring of Zookeeper processes. The change doesn't impact current installations and new installations requiring JMX remote monitoring need to set the jmx port to enable it. 0|i1x7un:
ZooKeeper ZOOKEEPER-1947

Implement a better way to specify joiners

Improvement Open Major Unresolved Unassigned Alexander Shraer Alexander Shraer 26/Jun/14 14:48   26/Jun/14 14:48       quorum, server   0 4   Currently a server must appear in its own config when it starts up. One of the reasons is that the server spec has LE ports through which it can connect to the leader. This means that when creating an initial configuration file of a server we'd like to add to the ensemble, we have to specify an invalid config where the new server already appears. This config is different from the current config and potentially from the new config that may eventually be installed.

Besides being a bogus config, this method means we have to be careful when adding multiple servers to the ensemble. If, for example, the current config is (A, B, C) and we'd like to add D and E, server D can have the initial config of (A, B, C, D) and server E the config (A, B, C, E) but not (A, B, C, D, E) since this risks C, D, E forming a quorum and loosing data (suppose that C was initially down and now C, D, E don't know the state of A, B and don't know that their own config is bogus). To see why its risky consider the indistinguishable case where A, B, C, D, E are all just starting from scratch and A and B are down.

One cleaner way to implement this would be to mark somehow the config rows corresponding to joining servers to indicate that they are not part of the config. Then take this information into account during leader election so that the joining servers can't vote to elect a leader. Such as server.5.joining=...

There are probably other good options to address the issue.
402122 No Perforce job exists for this issue. 0 402188
5 years, 39 weeks ago 0|i1x7qn:
ZooKeeper ZOOKEEPER-1946

Server logging should reflect dynamically reconfigured address

Improvement Resolved Minor Fixed Niko Vuokko Niko Vuokko Niko Vuokko 25/Jun/14 01:34   03/Jul/14 07:23 03/Jul/14 00:02 3.5.0 3.5.0 server   0 5   The server's client address:port is part of the QuorumPeer's thread name and thus shown in logs. Thread name is not currently updated after dynamic reconfiguration of the address, resulting in confusing log entries. patch 401748 No Perforce job exists for this issue. 3 401817
5 years, 38 weeks ago
Reviewed
0|i1x5g7:
ZooKeeper ZOOKEEPER-1945

deb - zkCli.sh, zkServer.sh and zkEnv.sh regression caused by ZOOKEEPER-1663

Bug Resolved Major Fixed Mark Flickinger Mark Flickinger Mark Flickinger 23/Jun/14 23:51   26/Jun/14 07:19 25/Jun/14 09:42 3.4.6 3.4.7, 3.5.0     0 4   Linux (Debian 7) with dash shell This is the same issue as ZOOKEEPER-1719, where the shebang for etc/init.d/zookeeper is set to /bin/sh, but needs to be fixed for deb packages(in src/packages/deb/init.d/zookeeper) 401456 No Perforce job exists for this issue. 1 401529
5 years, 39 weeks ago 0|i1x3of:
ZooKeeper ZOOKEEPER-1944

The 2pc picture of zookeeperInternals.pdf may be not correct.

Improvement Open Major Unresolved Unassigned zhangshuxing zhangshuxing 21/Jun/14 03:00   21/Jun/14 22:44   3.4.5   documentation   0 2   In the page 6, the 2pc diagram may be not right.The commit direction may be from leader to followers. 401131 No Perforce job exists for this issue. 0 401210
5 years, 39 weeks, 4 days ago 0|i1x1pz:
ZooKeeper ZOOKEEPER-1943

"src/contrib/zooinspector/NOTICE.txt" isn't complying to ".gitattributes" in branch-3.4

Bug Resolved Major Fixed Hongchao Deng Hongchao Deng Hongchao Deng 19/Jun/14 13:36   22/Jun/14 20:08 22/Jun/14 20:07 3.4.6 3.4.7 contrib   0 3   Using the git repo and checkout "branch-3.4", the "src/contrib/zooinspector/NOTICE.txt" file always shows up for changes but cannot be checkout or reset.

This is caused by ".gitattributes" line "text=auto" where git automatically sets the line ending stuff.

To solve this, I am gonna commit the git auto change and submit it as a patch for "branch-3.4".
400734 No Perforce job exists for this issue. 1 400828
5 years, 39 weeks, 4 days ago
Reviewed
0|i1wzd3:
ZooKeeper ZOOKEEPER-1942

ZooKeeper OSGi package imports: org.ietf.jgss dependency missing from manifest

Bug Resolved Major Duplicate Unassigned Kalvin Misquith Kalvin Misquith 18/Jun/14 10:48   12/Nov/15 15:06 21/Aug/15 09:24 3.4.6       7 11   For OSGI applications, the zookeeper manifest file should have org.ietf.jgss in its Import-Package statement. org.apache.zookeeper.client.ZooKeeperSaslClient imports org.ietf.jgss.*.

The following ClassDefNotFoundError occurs without it.

java.lang.NoClassDefFoundError: org.ietf.jgss.GSSException
at java.lang.J9VMInternals.verifyImpl(Native Method)
at java.lang.J9VMInternals.verify(J9VMInternals.java:94)
at java.lang.J9VMInternals.initialize(J9VMInternals.java:171)
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:945)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
Caused by: java.lang.ClassNotFoundException: org.ietf.jgss.GSSException
at org.eclipse.osgi.internal.loader.BundleLoader.findClassInternal(BundleLoader.java:501)
at org.eclipse.osgi.internal.loader.BundleLoader.findClass(BundleLoader.java:421)
at org.eclipse.osgi.internal.loader.BundleLoader.findClass(BundleLoader.java:412)
at org.eclipse.osgi.internal.baseadaptor.DefaultClassLoader.loadClass(DefaultClassLoader.java:107)
at java.lang.ClassLoader.loadClass(ClassLoader.java:707)
... 5 more
400425 No Perforce job exists for this issue. 0 400524
4 years, 19 weeks ago OSGI 0|i1wxhz:
ZooKeeper ZOOKEEPER-1941

2014-06-14 12:40:46,886 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server Will not attempt to authenticate using SASL (unknown error)

Improvement Open Major Unresolved Unassigned TIRUMALARAO KONDISETTY TIRUMALARAO KONDISETTY 14/Jun/14 13:52   14/Jun/14 13:52   3.4.6   java client   0 2   linux x86-64 399457 No Perforce job exists for this issue. 0 399566
5 years, 40 weeks, 5 days ago 0|i1wrlb:
ZooKeeper ZOOKEEPER-1940

Integrate with Docker.

Wish Open Trivial Unresolved Unassigned David Medinets David Medinets 13/Jun/14 12:57   01/Sep/19 02:40           1 7   Docker is an open platform for developers and sysadmins to build, ship, and run distributed applications.

It's become quite popular and I'd like to see the zookeeper community suggest a standard way to run zookeeper inside docker containers. To get the conversation started, I have a working example at:

https://github.com/medined/docker-zookeeper

I hope there is a better technique that I used. And if there is please make suggestions.

The difficulty, I think, posed by Docker, is that the images are started before the bridge network is created. This means, again I think, that zookeeper is running inside the container with no way to communicate with the ensemble for some non-trivial amount of time.

My resolution to this was to force each each to wait 30 seconds before starting zookeeper. I still see connection errors in the logs, but eventually the cluster settles and everything seems to work.

I'm hoping that someone which more networking experience than I can find a way to eliminate that 30 second delay and the connection errors during startup.

Thanks for reading this far.
399344 No Perforce job exists for this issue. 0 399453
28 weeks, 4 days ago 0|i1wqwv:
ZooKeeper ZOOKEEPER-1939

ReconfigRecoveryTest.testNextConfigUnreachable is failing

Bug Resolved Major Fixed Rakesh Radhakrishnan Rakesh Radhakrishnan Rakesh Radhakrishnan 12/Jun/14 02:39   26/Jun/14 07:19 25/Jun/14 13:29   3.4.7, 3.5.0 tests   0 4   Following is the failure log message:

2014-06-11 23:53:22,538 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@62] - TEST METHOD FAILED testNextConfigUnreachable
java.lang.AssertionError: QP failed to shutdown in 30 seconds: QuorumPeer[myid=0]/127.0.0.1:11251
at org.junit.Assert.fail(Assert.java:93)
at org.apache.zookeeper.test.QuorumBase.shutdown(QuorumBase.java:393)
at org.apache.zookeeper.server.quorum.QuorumPeerTestBase$TestQPMain.shutdown(QuorumPeerTestBase.java:52)
at org.apache.zookeeper.server.quorum.QuorumPeerTestBase$MainThread.shutdown(QuorumPeerTestBase.java:161)
at org.apache.zookeeper.server.quorum.ReconfigRecoveryTest.testNextConfigUnreachable(ReconfigRecoveryTest.java:268)
399007 No Perforce job exists for this issue. 1 399124
5 years, 39 weeks ago
Reviewed
0|i1wow7:
ZooKeeper ZOOKEEPER-1938

bump version in the C library as we prepare for 3.5.0 release

Task Resolved Trivial Fixed Raúl Gutiérrez Segalés Raúl Gutiérrez Segalés Raúl Gutiérrez Segalés 12/Jun/14 01:04   26/Jun/14 07:19 25/Jun/14 13:36 3.5.0 3.5.0 c client   0 5   building the C library from trunk now has 3.4.0 everywhere which makes it confusing when testing, so lets bump the version. 398994 No Perforce job exists for this issue. 2 399111
5 years, 39 weeks ago
Reviewed
0|i1wotb:
ZooKeeper ZOOKEEPER-1937

init script needs fixing for ZOOKEEPER-1719

Bug Resolved Major Fixed Marshall McMullen Nathan Sullivan Nathan Sullivan 10/Jun/14 23:28   25/Jul/14 07:25 24/Jul/14 19:17 3.4.6       0 7   Linux (Ubuntu 12.04) ZOOKEEPER-1719 changed the interpreter to bash for zkCli.sh, zkServer.sh and zkEnv.sh, but did not change src/packages/deb/init.d/zookeeper

This causes the following failure using /bin/sh

[...] root@hostname:~# service zookeeper stop
/etc/init.d/zookeeper: 81: /usr/libexec/zkEnv.sh: Syntax error: "(" unexpected (expecting "fi")

Simple fix, change the shebang to #!/bin/bash - tested and works fine.
398701 No Perforce job exists for this issue. 2 398825
5 years, 34 weeks, 6 days ago
Reviewed
0|i1wn33:
ZooKeeper ZOOKEEPER-1936

Server exits when unable to create data directory due to race

Bug Resolved Minor Fixed Ted Yu Harald Musum Harald Musum 10/Jun/14 03:25   24/Jan/20 09:20 23/Jan/20 10:47 3.4.6, 3.5.0 3.6.0 server   3 16 0 4200   We sometime see issues with ZooKeeper server not starting and seeing this error in the log:


[2014-05-27 09:29:48.248] ERROR : -
.org.apache.zookeeper.server.ZooKeeperServerMain Unexpected exception,
exiting abnormally\nexception=\njava.io.IOException: Unable to create data
directory /home/y/var/zookeeper/version-2\n\tat
org.apache.zookeeper.server.persistence.FileTxnSnapLog.<init>(FileTxnSnapLog.java:85)\n\tat
org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:103)\n\tat
org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:86)\n\tat
org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:52)\n\tat
org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:116)\n\tat
org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:78)\n\t
[...]

Stack trace from JVM gives this:

"PurgeTask" daemon prio=10 tid=0x000000000201d000 nid=0x1727 runnable
[0x00007f55d7dc7000]
java.lang.Thread.State: RUNNABLE
at java.io.UnixFileSystem.createDirectory(Native Method)
at java.io.File.mkdir(File.java:1310)
at java.io.File.mkdirs(File.java:1337)
at
org.apache.zookeeper.server.persistence.FileTxnSnapLog.<init>(FileTxnSnapLog.java:84)
at org.apache.zookeeper.server.PurgeTxnLog.purge(PurgeTxnLog.java:68)
at
org.apache.zookeeper.server.DatadirCleanupManager$PurgeTask.run(DatadirCleanupManager.java:140)
at java.util.TimerThread.mainLoop(Timer.java:555)
at java.util.TimerThread.run(Timer.java:505)

"zookeeper server" prio=10 tid=0x00000000027df800 nid=0x1715 runnable
[0x00007f55d7ed8000]
java.lang.Thread.State: RUNNABLE
at java.io.UnixFileSystem.createDirectory(Native Method)
at java.io.File.mkdir(File.java:1310)
at java.io.File.mkdirs(File.java:1337)
at
org.apache.zookeeper.server.persistence.FileTxnSnapLog.<init>(FileTxnSnapLog.java:84)
at
org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:103)
at
org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:86)
at
org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:52)
at
org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:116)
at
org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:78)
[...]

So it seems that when autopurge is used (as it is in our case), it might happen at the same time as starting the server itself. In FileTxnSnapLog() it will check if the directory exists and create it if not. These two tasks do this at the same time, and mkdir fails and server exits the JVM.

100% 100% 4200 0 pull-request-available 398490 No Perforce job exists for this issue. 7 398615
8 weeks ago 0|i1wlsf:
ZooKeeper ZOOKEEPER-1935

Current ZooKeeper C Library (Client) makes a distinction between removing a child node and adding a child node on receiving ZOO_CHILD_EVENT session event

Improvement Open Major Unresolved Unassigned Fuyan Wang Fuyan Wang 10/Jun/14 01:46   10/Jun/14 01:46       c client   0 1   I want to determine whether a node adds or removes a child node. But based on the currently implementation, I can only to get the session event ZOO_CHILD_EVENT. There is no ways to determine whether it is an add child event or a delete child event.

If we enhance this event, it can benefit people having a similar requirement. Thanks.
398474 No Perforce job exists for this issue. 0 398599
5 years, 41 weeks, 2 days ago 0|i1wlov:
ZooKeeper ZOOKEEPER-1934

Stale data received from sync'd ensemble peer

Bug Open Major Unresolved Unassigned Marshall McMullen Marshall McMullen 05/Jun/14 19:01   16/Sep/14 12:04   3.5.0       0 5   In our regression testing we encountered an error wherein we were caching a value we read from zookeeper and then experienced session loss. We subsequently got reconnected to a different zookeeper server. When we tried to read the same path from this new zookeeper server we are getting a stale value.

Specifically, we are reading "/binchanges" and originally got back a value of "3" from the first server. After we lost connection and reconnected before the session timeout, we then read "/binchanges" from the new server and got back a value of "2". In our code path we never set this value from 3 to 2. We throw an assertion if the value ever goes backwards. Which is how we caught this error.

It's my understanding of the single system image guarantee that this should never be allowed. I realize that the single system image guarantee is still quorum based and it's certainly possible that a minority of the ensemble may have stale data. However, I also believe that each client has to send the highest zxid it's seen as part of its connection request to the server. And if the server it's connecting to has a smaller zxid than the value the client sends, then the connection request should be refused.

Assuming I have all of that correct, then I'm at a loss for how this happened.

The failure happened around Jun 4 08:13:44. Just before that, at June 4 08:13:30 there was a round of leader election. During that round of leader election we voted server with id=4 and zxid=0x300001c4c. This then led to a new zxid=0x400000001. The new leader sends a diff to all the servers including the one we will soon read the stale data from (id=2). Server with ID=2's log files also reflect that as of 08:13:43 it was up to date and current with an UPTODATE message.

I'm going to attach log files from all 5 ensemble nodes. I also used zktreeutil to dump the database out for the 5 ensemble nodes. I diff'd those, and compared them all for correctness. 1 of the nodes (id=2) has a massively divergent zktreeutil dump than the other 4 nodes even though it received the diff from the new leader.

In the attachments there are 5 nodes. I will number each log file by it's zookeeper id, e.g. node4.log.



396939 No Perforce job exists for this issue. 5 397057
5 years, 27 weeks, 2 days ago 0|i1wc6f:
ZooKeeper ZOOKEEPER-1933

Windows release build of zk client cannot connect to zk server

Bug Resolved Major Fixed Orion Hodson Norris Lee Norris Lee 03/Jun/14 12:59   25/Jul/14 13:39 25/Jul/14 12:58 3.4.6 3.5.0 c client   1 7   When building zookeeper in Visual Studio in debug mode, the client can connect to the server without error. When building in release mode, I get a continuous error message:
{code}
2014-06-02 11:25:20,070:7144(0xc84):ZOO_INFO@zookeeper_init_internal@1008: Initiating client connection, host=192.168.39.43:5181 sessionTimeout=30000 watcher=10049C90 sessionId=0 sessionPasswd=<null> context=001FC0F0 flags=0
2014-06-02 11:25:20,072:7144(0xc84):ZOO_DEBUG@start_threads@221: starting threads...
2014-06-02 11:25:20,072:7144(0x1ea0):ZOO_DEBUG@do_completion@460: started completion thread
2014-06-02 11:25:20,072:7144(0x1e08):ZOO_DEBUG@do_io@403: started IO thread
2014-06-02 11:25:20,072:7144(0x1e08):ZOO_DEBUG@get_next_server_in_reconfig@1148: [OLD] count=0 capacity=0 next=0 hasnext=0
2014-06-02 11:25:20,072:7144(0x1e08):ZOO_DEBUG@get_next_server_in_reconfig@1151: [NEW] count=1 capacity=16 next=0 hasnext=1
2014-06-02 11:25:20,072:7144(0x1e08):ZOO_DEBUG@get_next_server_in_reconfig@1160: Using next from NEW=192.168.39.43:5181
2014-06-02 11:25:20,072:7144(0x1e08):ZOO_DEBUG@zookeeper_interest@1992: [zk] connect()

2014-06-02 11:25:20,158:7144(0x1e08):ZOO_ERROR@handle_socket_error_msg@1847: Socket [192.168.39.43:5181] zk retcode=-4, errno=10035(Unknown error): failed to send a handshake packet: Unknown error
2014-06-02 11:25:20,158:7144(0x1e08):ZOO_DEBUG@handle_error@1595: Previous connection=[192.168.39.43:5181] delay=0
2014-06-02 11:25:20,158:7144(0x1e08):ZOO_DEBUG@get_next_server_in_reconfig@1148: [OLD] count=0 capacity=0 next=0 hasnext=0
2014-06-02 11:25:20,158:7144(0x1e08):ZOO_DEBUG@get_next_server_in_reconfig@1151: [NEW] count=1 capacity=16 next=0 hasnext=1
2014-06-02 11:25:20,158:7144(0x1e08):ZOO_DEBUG@get_next_server_in_reconfig@1160: Using next from NEW=192.168.39.43:5181
2014-06-02 11:25:20,158:7144(0x1e08):ZOO_DEBUG@zookeeper_interest@1992: [zk] connect()

2014-06-02 11:25:20,159:7144(0x1e08):ZOO_ERROR@handle_socket_error_msg@1847: Socket [192.168.39.43:5181] zk retcode=-4, errno=10035(Unknown error): failed to send a handshake packet: Unknown error
2014-06-02 11:25:20,159:7144(0x1e08):ZOO_DEBUG@handle_error@1595: Previous connection=[192.168.39.43:5181] delay=0
2014-06-02 11:25:20,159:7144(0x1e08):ZOO_DEBUG@get_next_server_in_reconfig@1148: [OLD] count=0 capacity=0 next=0 hasnext=0
2014-06-02 11:25:20,159:7144(0x1e08):ZOO_DEBUG@get_next_server_in_reconfig@1151: [NEW] count=1 capacity=16 next=0 hasnext=1
2014-06-02 11:25:20,159:7144(0x1e08):ZOO_DEBUG@get_next_server_in_reconfig@1160: Using next from NEW=192.168.39.43:5181
2014-06-02 11:25:20,159:7144(0x1e08):ZOO_DEBUG@zookeeper_interest@1992: [zk] connect()

2014-06-02 11:25:20,159:7144(0x1e08):ZOO_ERROR@handle_socket_error_msg@1847: Socket [192.168.39.43:5181] zk retcode=-4, errno=10035(Unknown error): failed to send a handshake packet: Unknown error
2014-06-02 11:25:20,159:7144(0x1e08):ZOO_DEBUG@handle_error@1595: Previous connection=[192.168.39.43:5181] delay=0
2014-06-02 11:25:20,159:7144(0x1e08):ZOO_DEBUG@get_next_server_in_reconfig@1148: [OLD] count=0 capacity=0 next=0 hasnext=0
2014-06-02 11:25:20,159:7144(0x1e08):ZOO_DEBUG@get_next_server_in_reconfig@1151: [NEW] count=1 capacity=16 next=0 hasnext=1
2014-06-02 11:25:20,159:7144(0x1e08):ZOO_DEBUG@get_next_server_in_reconfig@1160: Using next from NEW=192.168.39.43:5181
2014-06-02 11:25:20,159:7144(0x1e08):ZOO_DEBUG@zookeeper_interest@1992: [zk] connect()
{code}
396397 No Perforce job exists for this issue. 8 396519
5 years, 34 weeks, 6 days ago
Reviewed
0|i1w8v3:
ZooKeeper ZOOKEEPER-1932

Remove deprecated LeaderElection class

Bug Resolved Major Fixed Michael Han Michi Mutsuzaki Michi Mutsuzaki 01/Jun/14 20:58   11/May/17 12:42 11/May/17 11:01 3.5.0 3.6.0 leaderElection   0 7   org.apache.zookeeper.test.LETest.testLE is failing on trunk once in a while. I'm not able to reproduce the failure on my box. I looked at the log, but I couldn't quite figure out what's going on.

https://builds.apache.org/view/S-Z/view/ZooKeeper/job/ZooKeeper-trunk/2315/testReport/

Update:
======
Because LE is deprecated there is not much points on spending effort fixing it, as discussed in the JIRA. Updated JIRA title to reflect the state of the issue.
396034 No Perforce job exists for this issue. 3 396160
2 years, 45 weeks ago 0|i1w6nr:
ZooKeeper ZOOKEEPER-1931

intern project idea: implement zab

New Feature Open Major Unresolved Michi Mutsuzaki Michi Mutsuzaki Michi Mutsuzaki 01/Jun/14 16:09   14/Jan/16 20:12           0 6   The goal of this project is to define an interface for replication protocol and implement the interface using ZAB. This project will most likely be done outside of ZooKeeper to avoid impacting the stability of the ZooKeeper code base, but I'm opening a JIRA here to gauge interest and get feedback from ZooKeeper community.

There are 2 main motivations for this project:

1. There are many use cases that need a replication protocol like ZAB, but ZooKeeper's hierarchical data model doesn't work well. It's difficult to use ZAB without ZooKeeper with the way ZooKeeper code is currently structured.
2. It's valuable to have a common interface for replication protocol to build services on. This allows you to plug in different implementations for benchmarking and testing for correctness. This point is related to ZOOKEEPER-30.

The project is roughly broken into 4 pieces:

1. Define the interface for replication protocol. It's very important to get the interface right. I'd appreciate if you guys can help define the interface.
2. Implement the interface with single node ZAB.
3. Implement a simple reference service, something like a key-value store or a benchmark tool.
4. Implement ZAB, either from scratch or by refactoring / curving off unnecessary parts from the ZooKeeper code base.

I have some questions:

- How do things like session tracker and dynamic reconfiguration fit into this? Should they be separate optional interfaces?
- Where should this project belong to? Is it worth making this an incubator project, or should I just put the code on github? I'd like to make it easy for people from different organizations to collaborate (in terms of license grant and all) from the beginning.
395993 No Perforce job exists for this issue. 0 396119
5 years, 41 weeks, 1 day ago 0|i1w6en:
ZooKeeper ZOOKEEPER-1930

A typo in zookeeper recipes.html

Bug Resolved Minor Fixed Chengwei Yang Chengwei Yang Chengwei Yang 28/May/14 01:47   29/May/14 07:35 28/May/14 13:58     documentation   0 4   svn trunk repo, revision 1597694 With the sequence flag, ZooKeeper automatically appends a sequence number that is *greater that* any one previously appended to a child of "/election".

*greater that* above should be *greater than*
395184 No Perforce job exists for this issue. 1 395315
5 years, 43 weeks ago 0|i1w1gf:
ZooKeeper ZOOKEEPER-1929

std::length_error on update children

Bug Closed Major Fixed Charles Strahan Eduard White Eduard White 24/May/14 00:13   21/Jul/16 16:18 21/Nov/15 15:15 3.3.5, 3.4.6, 3.5.1 3.4.8, 3.5.2 contrib-zkfuse   0 5   debian Trying to open zk root directory:

./zkfuse -z localhost:2181 -m /CLOUD/zookeeper -d
1 [0x7f89362d0780] INFO zkfuse null - Starting zkfuse
cacheSize = 256, debug = 1, forceDirSuffix = "._dir_", mount = "/CLOUD/zookeeper", name = "_data_", zookeeper = "localhost:2181", optind = 6, argc = 6, current arg = "NULL"
1 [0x7f89362d0780] INFO zkfuse null - Create ZK adapter
1 [0x7f89362d0780] TRACE zookeeper.adapter null - zk::ZooKeeperAdapter::ZooKeeperAdapter(zk::ZooKeeperConfig, zk::ZKEventListener*, bool)::Trace::Trace(const void*) 0x434ecd Enter
1 [0x7f89362d0780] TRACE zookeeper.adapter null - zk::ZooKeeperAdapter::ZooKeeperAdapter(zk::ZooKeeperConfig, zk::ZKEventListener*, bool)::Trace::~Trace() 0x434ecd Exit
1 [0x7f89362bd700] TRACE zookeeper.adapter null - zk::ZooKeeperAdapter::processEvents()::Trace::Trace(const void*) 0x434df4 Enter
1 [0x7f89327bb700] TRACE zookeeper.adapter null - zk::ZooKeeperAdapter::processUserEvents()::Trace::Trace(const void*) 0x434e60 Enter
1 [0x7f89362d0780] TRACE zookeeper.adapter null - zk::ZooKeeperAdapter::reconnect()::Trace::Trace(const void*) 0x434c71 Enter
1 [0x7f89362d0780] TRACE zookeeper.adapter null - zk::ZooKeeperAdapter::disconnect()::Trace::Trace(const void*) 0x434c4c Enter
1 [0x7f89362d0780] TRACE zookeeper.adapter null - mp_zkHandle: (nil), state 0
1 [0x7f89362d0780] TRACE zookeeper.adapter null - zk::ZooKeeperAdapter::disconnect()::Trace::~Trace() 0x434c4c Exit
2014-05-24 08:07:44,860:20540(0x7f89362d0780):ZOO_INFO@log_env@712: Client environment:zookeeper.version=zookeeper C client 3.4.6
2014-05-24 08:07:44,860:20540(0x7f89362d0780):ZOO_INFO@log_env@716: Client environment:host.name=nanoha
2014-05-24 08:07:44,860:20540(0x7f89362d0780):ZOO_INFO@log_env@723: Client environment:os.name=Linux
2014-05-24 08:07:44,860:20540(0x7f89362d0780):ZOO_INFO@log_env@724: Client environment:os.arch=3.2.0-4-amd64
2014-05-24 08:07:44,860:20540(0x7f89362d0780):ZOO_INFO@log_env@725: Client environment:os.version=#1 SMP Debian 3.2.54-2
2014-05-24 08:07:44,860:20540(0x7f89362d0780):ZOO_INFO@log_env@733: Client environment:user.name=root
2014-05-24 08:07:44,860:20540(0x7f89362d0780):ZOO_INFO@log_env@741: Client environment:user.home=/root
2014-05-24 08:07:44,860:20540(0x7f89362d0780):ZOO_INFO@log_env@753: Client environment:user.dir=/opt/zoo/3.4.6/build/contrib/zkfuse/src
2014-05-24 08:07:44,860:20540(0x7f89362d0780):ZOO_INFO@zookeeper_init@786: Initiating client connection, host=localhost:2181 sessionTimeout=1000 watcher=0x429780 sessionId=0 sessionPasswd=<null> context=0x9b5e30 flags=0
2014-05-24 08:07:44,861:20540(0x7f89362d0780):ZOO_DEBUG@start_threads@221: starting threads...
2 [0x7f89362d0780] TRACE zookeeper.adapter null - zk::ZooKeeperAdapter::setState(zk::ZooKeeperAdapter::AdapterState)::Trace::Trace(const void*) 0x434c43 Enter
2 [0x7f89362d0780] INFO zookeeper.adapter null - Adapter state transition: 0 -> 1
2014-05-24 08:07:44,861:20540(0x7f8931899700):ZOO_DEBUG@do_completion@459: started completion thread
2 [0x7f89362d0780] TRACE zookeeper.adapter null - zk::ZooKeeperAdapter::setState(zk::ZooKeeperAdapter::AdapterState)::Trace::~Trace() 0x434c43 Exit
3 [0x7f89362d0780] DEBUG zookeeper.adapter null - mp_zkHandle: 0x9bbba0, state 1
2014-05-24 08:07:44,861:20540(0x7f893209a700):ZOO_DEBUG@do_io@367: started IO thread
3 [0x7f89362d0780] TRACE zookeeper.adapter null - zk::ZooKeeperAdapter::reconnect()::Trace::~Trace() 0x434c71 Exit
ZOOKEEPER_ROOT_CHILDREN_WATCH_BUG enabled
3 [0x7f89362d0780] INFO zkfuse null - Initialize fuse
2014-05-24 08:07:44,861:20540(0x7f893209a700):ZOO_INFO@check_events@1705: initiated connection to server [127.0.0.1:2181]
FUSE library version: 2.9.3
nullpath_ok: 0
nopath: 0
utime_omit_ok: 0
unique: 1, opcode: INIT (26), nodeid: 0, insize: 56, pid: 0
INIT: 7.17
flags=0x0000047b
max_readahead=0x00020000
INIT: 7.19
flags=0x00000013
max_readahead=0x00020000
max_write=0x00020000
max_background=0
congestion_threshold=0
unique: 1, success, outsize: 40
2014-05-24 08:07:44,909:20540(0x7f893209a700):ZOO_INFO@check_events@1752: session establishment complete on server [127.0.0.1:2181], sessionId=0x1461f2be1b10025, negotiated timeout=4000
2014-05-24 08:07:44,909:20540(0x7f893209a700):ZOO_DEBUG@check_events@1758: Calling a watcher for a ZOO_SESSION_EVENT and the state=ZOO_CONNECTED_STATE
2014-05-24 08:07:44,910:20540(0x7f8931899700):ZOO_DEBUG@process_completions@2113: Calling a watcher for node [], type = -1 event=ZOO_SESSION_EVENT
51 [0x7f8931899700] TRACE zookeeper.adapter null - zk::zkWatcher(zhandle_t*, int, int, const char*, void*)::Trace::Trace(const void*) 0x434e10 Enter
51 [0x7f8931899700] INFO zookeeper.adapter null - Received a ZK event - type: -1, state: 3, path: ''
51 [0x7f8931899700] TRACE zookeeper.adapter null - zk::ZooKeeperAdapter::enqueueEvent(int, int, const string&)::Trace::Trace(const void*) 0x434e02 Enter
51 [0x7f8931899700] TRACE zookeeper.adapter null - zk::ZooKeeperAdapter::enqueueEvent(int, int, const string&)::Trace::~Trace() 0x434e02 Exit
51 [0x7f8931899700] TRACE zookeeper.adapter null - zk::zkWatcher(zhandle_t*, int, int, const char*, void*)::Trace::~Trace() 0x434e10 Exit
51 [0x7f89362bd700] INFO zookeeper.adapter null - Received SESSION event, state: 3. Adapter state: 1
51 [0x7f89362bd700] TRACE zookeeper.adapter null - zk::ZooKeeperAdapter::setState(zk::ZooKeeperAdapter::AdapterState)::Trace::Trace(const void*) 0x434c43 Enter
51 [0x7f89362bd700] INFO zookeeper.adapter null - Adapter state transition: 1 -> 2
51 [0x7f89362bd700] TRACE zookeeper.adapter null - zk::ZooKeeperAdapter::setState(zk::ZooKeeperAdapter::AdapterState)::Trace::~Trace() 0x434c43 Exit
52 [0x7f89327bb700] TRACE zookeeper.adapter null - zk::ZooKeeperAdapter::handleEvent(int, int, const string&)::Trace::Trace(const void*) 0x434e37 Enter
52 [0x7f89327bb700] TRACE zookeeper.adapter null - type: -1, state 3, path:
52 [0x7f89327bb700] TRACE zookeeper.adapter null - zk::ZooKeeperAdapter::handleEvent(int, int, const string&, const Listener2Context&)::Trace::Trace(const void*) 0x434de7 Enter
52 [0x7f89327bb700] DEBUG zkfuse null - eventReceived() eventType -1, eventState 3, path
52 [0x7f89327bb700] TRACE zkfuse null - *** CONNECTED ***
52 [0x7f89327bb700] TRACE zookeeper.adapter null - zk::ZooKeeperAdapter::handleEvent(int, int, const string&, const Listener2Context&)::Trace::~Trace() 0x434de7 Exit
52 [0x7f89327bb700] TRACE zookeeper.adapter null - zk::ZooKeeperAdapter::handleEvent(int, int, const string&)::Trace::~Trace() 0x434e37 Exit
2014-05-24 08:07:46,196:20540(0x7f893209a700):ZOO_DEBUG@zookeeper_process@2264: Got ping response in 0 ms
2014-05-24 08:07:47,531:20540(0x7f893209a700):ZOO_DEBUG@zookeeper_process@2264: Got ping response in 0 ms
2014-05-24 08:07:48,865:20540(0x7f893209a700):ZOO_DEBUG@zookeeper_process@2264: Got ping response in 0 ms
2014-05-24 08:07:50,200:20540(0x7f893209a700):ZOO_DEBUG@zookeeper_process@2264: Got ping response in 0 ms
2014-05-24 08:07:51,535:20540(0x7f893209a700):ZOO_DEBUG@zookeeper_process@2264: Got ping response in 0 ms
2014-05-24 08:07:52,869:20540(0x7f893209a700):ZOO_DEBUG@zookeeper_process@2264: Got ping response in 0 ms
unique: 2, opcode: ACCESS (34), nodeid: 1, insize: 48, pid: 16822
unique: 2, error: -38 (Function not implemented), outsize: 16
unique: 3, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 16822
getattr /
8606 [0x7f8931098700] DEBUG zkfuse null - zkfuse_getattr(path /)
8606 [0x7f8931098700] DEBUG zkfuse null - getattr(path /)
8606 [0x7f8931098700] DEBUG zkfuse null - getZkPath(path /)
8606 [0x7f8931098700] DEBUG zkfuse null - getZkPath returns /, nameType 2
8606 [0x7f8931098700] DEBUG zkfuse null - open(path /, justCreated 0)
8606 [0x7f8931098700] DEBUG zkfuse null - allocate(path /)
8606 [0x7f8931098700] DEBUG zkfuse null - not found
8606 [0x7f8931098700] DEBUG zkfuse null - free list empty, resize handle 1
8606 [0x7f8931098700] DEBUG zkfuse null - constructor() path /
8606 [0x7f8931098700] DEBUG zkfuse null - incRefCount(count 0) path /
8606 [0x7f8931098700] DEBUG zkfuse null - incRefCount returns 1
8606 [0x7f8931098700] DEBUG zkfuse null - numInUse 1
8606 [0x7f8931098700] DEBUG zkfuse null - allocate returns 1, newFile 1
8606 [0x7f8931098700] DEBUG zkfuse null - update(newFile 1) path /
8606 [0x7f8931098700] DEBUG zkfuse null - initialized children 0, data 0
8606 [0x7f8931098700] DEBUG zkfuse null - has children watch 0, data watch 0
8606 [0x7f8931098700] DEBUG zkfuse null - update children
8606 [0x7f8931098700] TRACE zookeeper.adapter null - zk::ZooKeeperAdapter::getNodeChildren(std::vector<std::basic_string<char> >&, const string&, zk::ZKEventListener*, void*)::Trace::Trace(const void*) 0x434ede Enter
8607 [0x7f8931098700] TRACE zookeeper.adapter null - zk::ZooKeeperAdapter::validatePath(const string&)::Trace::Trace(const void*) 0x434cc4 Enter
8607 [0x7f8931098700] TRACE zookeeper.adapter null - zk::ZooKeeperAdapter::validatePath(const string&)::Trace::~Trace() 0x434cc4 Exit
8607 [0x7f8931098700] TRACE zookeeper.adapter null - zk::ZooKeeperAdapter::verifyConnection()::Trace::Trace(const void*) 0x434cb3 Enter
8607 [0x7f8931098700] TRACE zookeeper.adapter null - zk::ZooKeeperAdapter::verifyConnection()::Trace::~Trace() 0x434cb3 Exit
2014-05-24 08:07:53,465:20540(0x7f8931098700):ZOO_DEBUG@zoo_awget_children_@2874: Sending request xid=0x53801b11 for path [/] to 127.0.0.1:2181
2014-05-24 08:07:53,465:20540(0x7f893209a700):ZOO_DEBUG@process_sync_completion@1870: Processing sync_completion with type=3 xid=0x53801b11 rc=0
8607 [0x7f8931098700] TRACE zookeeper.adapter null - zk::ZooKeeperAdapter::getNodeChildren(std::vector<std::basic_string<char> >&, const string&, zk::ZKEventListener*, void*)::Trace::~Trace() 0x434ede Exit
8607 [0x7f8931098700] DEBUG zkfuse null - update children done
8607 [0x7f8931098700] DEBUG zkfuse null - node first use or reuse
8607 [0x7f8931098700] DEBUG zkfuse null - update data
8607 [0x7f8931098700] TRACE zookeeper.adapter null - zk::ZooKeeperAdapter::getNodeData(const string&, zk::ZKEventListener*, void*, Stat*)::Trace::Trace(const void*) 0x434e82 Enter
8607 [0x7f8931098700] TRACE zookeeper.adapter null - zk::ZooKeeperAdapter::validatePath(const string&)::Trace::Trace(const void*) 0x434cc4 Enter
8607 [0x7f8931098700] TRACE zookeeper.adapter null - zk::ZooKeeperAdapter::validatePath(const string&)::Trace::~Trace() 0x434cc4 Exit
8607 [0x7f8931098700] TRACE zookeeper.adapter null - zk::ZooKeeperAdapter::verifyConnection()::Trace::Trace(const void*) 0x434cb3 Enter
8607 [0x7f8931098700] TRACE zookeeper.adapter null - zk::ZooKeeperAdapter::verifyConnection()::Trace::~Trace() 0x434cb3 Exit
2014-05-24 08:07:53,466:20540(0x7f8931098700):ZOO_DEBUG@zoo_awget@2661: Sending request xid=0x53801b12 for path [/] to 127.0.0.1:2181
2014-05-24 08:07:53,466:20540(0x7f893209a700):ZOO_DEBUG@process_sync_completion@1870: Processing sync_completion with type=2 xid=0x53801b12 rc=0
8608 [0x7f8931098700] TRACE zookeeper.adapter null - zk::ZooKeeperAdapter::getNodeData(const string&, zk::ZKEventListener*, void*, Stat*)::Trace::~Trace() 0x434e82 Exit
8608 [0x7f8931098700] DEBUG zkfuse null - update data done, latest version 0
8608 [0x7f8931098700] DEBUG zkfuse null - update set active version 0
8608 [0x7f8931098700] DEBUG zkfuse null - update returns 0
8608 [0x7f8931098700] DEBUG zkfuse null - open returns 1
8608 [0x7f8931098700] DEBUG zkfuse null - getattr(nameType 2) path /
8608 [0x7f8931098700] DEBUG zkfuse null - isRegNameType(nameType 2) returns 0
8608 [0x7f8931098700] DEBUG zkfuse null - directory
8608 [0x7f8931098700] DEBUG zkfuse null - hasChild(childPath /.zkfuse.dir) returns 0
8608 [0x7f8931098700] DEBUG zkfuse null - getattr returns 0
8608 [0x7f8931098700] DEBUG zkfuse null - close() path /
8608 [0x7f8931098700] DEBUG zkfuse null - flush() path /
8608 [0x7f8931098700] DEBUG zkfuse null - not dirty
8608 [0x7f8931098700] DEBUG zkfuse null - flush returns 0
8608 [0x7f8931098700] DEBUG zkfuse null - deallocate(handle 1)
8608 [0x7f8931098700] DEBUG zkfuse null - incRefCount(count -1) path /
8608 [0x7f8931098700] DEBUG zkfuse null - incRefCount returns 0
8608 [0x7f8931098700] DEBUG zkfuse null - path / ref count 0
8608 [0x7f8931098700] DEBUG zkfuse null - deallocate done
8608 [0x7f8931098700] DEBUG zkfuse null - close returns 0
8608 [0x7f8931098700] DEBUG zkfuse null - getattr returns 0
8608 [0x7f8931098700] DEBUG zkfuse null - zkfuse_getattr returns 0
unique: 3, success, outsize: 120
unique: 4, opcode: OPENDIR (27), nodeid: 1, insize: 48, pid: 16822
opendir flags: 0x98800 /
unique: 5, opcode: INTERRUPT (36), nodeid: 0, insize: 48, pid: 0
8610 [0x7f8930897700] DEBUG zkfuse null - zkfuse_opendir(path /)
INTERRUPT: 4
8610 [0x7f8930897700] DEBUG zkfuse null - getZkPath(path /)
8610 [0x7f8930897700] DEBUG zkfuse null - getZkPath returns /, nameType 2
8610 [0x7f8930897700] DEBUG zkfuse null - open(path /, justCreated 0)
8610 [0x7f8930897700] DEBUG zkfuse null - allocate(path /)
8610 [0x7f8930897700] DEBUG zkfuse null - found
8610 [0x7f8930897700] DEBUG zkfuse null - incRefCount(count 1) path /
8610 [0x7f8930897700] DEBUG zkfuse null - incRefCount returns 1
8610 [0x7f8930897700] DEBUG zkfuse null - resurrecting zombie, numInUse 1
8610 [0x7f8930897700] DEBUG zkfuse null - allocate returns 1, newFile 0
8611 [0x7f8930897700] DEBUG zkfuse null - update(newFile 0) path /
8611 [0x7f8930897700] DEBUG zkfuse null - initialized children 1, data 1
8611 [0x7f8930897700] DEBUG zkfuse null - has children watch 1, data watch 1
8611 [0x7f8930897700] DEBUG zkfuse null - update children
8611 [0x7f8930897700] TRACE zookeeper.adapter null - zk::ZooKeeperAdapter::getNodeChildren(std::vector<std::basic_string<char> >&, const string&, zk::ZKEventListener*, void*)::Trace::Trace(const void*) 0x434ede Enter
8611 [0x7f8930897700] TRACE zookeeper.adapter null - zk::ZooKeeperAdapter::validatePath(const string&)::Trace::Trace(const void*) 0x434cc4 Enter
8611 [0x7f8930897700] TRACE zookeeper.adapter null - zk::ZooKeeperAdapter::validatePath(const string&)::Trace::~Trace() 0x434cc4 Exit
8611 [0x7f8930897700] TRACE zookeeper.adapter null - zk::ZooKeeperAdapter::verifyConnection()::Trace::Trace(const void*) 0x434cb3 Enter
8611 [0x7f8930897700] TRACE zookeeper.adapter null - zk::ZooKeeperAdapter::verifyConnection()::Trace::~Trace() 0x434cb3 Exit
2014-05-24 08:07:53,469:20540(0x7f8930897700):ZOO_DEBUG@zoo_awget_children_@2874: Sending request xid=0x53801b13 for path [/] to 127.0.0.1:2181
2014-05-24 08:07:53,469:20540(0x7f893209a700):ZOO_DEBUG@process_sync_completion@1870: Processing sync_completion with type=3 xid=0x53801b13 rc=0
8611 [0x7f8930897700] TRACE zookeeper.adapter null - zk::ZooKeeperAdapter::getNodeChildren(std::vector<std::basic_string<char> >&, const string&, zk::ZKEventListener*, void*)::Trace::~Trace() 0x434ede Exit
8611 [0x7f8930897700] DEBUG zkfuse null - update children done
8611 [0x7f8930897700] DEBUG zkfuse null - node first use or reuse
8611 [0x7f8930897700] DEBUG zkfuse null - update set active version 0
8611 [0x7f8930897700] DEBUG zkfuse null - update returns 0
8611 [0x7f8930897700] DEBUG zkfuse null - open returns 1
8611 [0x7f8930897700] DEBUG zkfuse null - incOpenDirCount(count 1) path /
8611 [0x7f8930897700] DEBUG zkfuse null - incOpenDirCount returns 1
8611 [0x7f8930897700] DEBUG zkfuse null - zkfuse_opendir returns 0
opendir[1] flags: 0x98800 /
unique: 4, success, outsize: 32
unique: 6, opcode: READDIR (28), nodeid: 1, insize: 80, pid: 16822
readdir[1] from 0
8612 [0x7f8913fff700] DEBUG zkfuse null - zkfuse_readdir(path /, offset 0)
8612 [0x7f8913fff700] DEBUG zkfuse null - readdir(offset 0) path /
8612 [0x7f8913fff700] DEBUG zkfuse null - isMeta(childName /aliases.json) returns 0
8612 [0x7f8913fff700] DEBUG zkfuse null - isMeta(childName /clusterstate.json) returns 0
8612 [0x7f8913fff700] DEBUG zkfuse null - isMeta(childName /collections) returns 0
8612 [0x7f8913fff700] DEBUG zkfuse null - isMeta(childName /configs) returns 0
8612 [0x7f8913fff700] DEBUG zkfuse null - isMeta(childName /live_nodes) returns 0
8612 [0x7f8913fff700] DEBUG zkfuse null - isMeta(childName /overseer) returns 0
8612 [0x7f8913fff700] DEBUG zkfuse null - isMeta(childName /overseer_elect) returns 0
8612 [0x7f8913fff700] DEBUG zkfuse null - isMeta(childName /zookeeper) returns 0
8612 [0x7f8913fff700] DEBUG zkfuse null - open(path /aliases.json, justCreated 0)
8612 [0x7f8913fff700] DEBUG zkfuse null - allocate(path /aliases.json)
8612 [0x7f8913fff700] DEBUG zkfuse null - not found
8612 [0x7f8913fff700] DEBUG zkfuse null - free list empty, resize handle 2
8612 [0x7f8913fff700] DEBUG zkfuse null - constructor() path /aliases.json
8612 [0x7f8913fff700] DEBUG zkfuse null - incRefCount(count 0) path /aliases.json
8612 [0x7f8913fff700] DEBUG zkfuse null - incRefCount returns 1
8612 [0x7f8913fff700] DEBUG zkfuse null - numInUse 2
8612 [0x7f8913fff700] DEBUG zkfuse null - allocate returns 2, newFile 1
8612 [0x7f8913fff700] DEBUG zkfuse null - update(newFile 1) path /aliases.json
8612 [0x7f8913fff700] DEBUG zkfuse null - initialized children 0, data 0
8612 [0x7f8913fff700] DEBUG zkfuse null - has children watch 0, data watch 0
8613 [0x7f8913fff700] DEBUG zkfuse null - update children
8613 [0x7f8913fff700] TRACE zookeeper.adapter null - zk::ZooKeeperAdapter::getNodeChildren(std::vector<std::basic_string<char> >&, const string&, zk::ZKEventListener*, void*)::Trace::Trace(const void*) 0x434ede Enter
8613 [0x7f8913fff700] TRACE zookeeper.adapter null - zk::ZooKeeperAdapter::validatePath(const string&)::Trace::Trace(const void*) 0x434cc4 Enter
8613 [0x7f8913fff700] TRACE zookeeper.adapter null - zk::ZooKeeperAdapter::validatePath(const string&)::Trace::~Trace() 0x434cc4 Exit
8613 [0x7f8913fff700] TRACE zookeeper.adapter null - zk::ZooKeeperAdapter::verifyConnection()::Trace::Trace(const void*) 0x434cb3 Enter
8613 [0x7f8913fff700] TRACE zookeeper.adapter null - zk::ZooKeeperAdapter::verifyConnection()::Trace::~Trace() 0x434cb3 Exit
2014-05-24 08:07:53,471:20540(0x7f8913fff700):ZOO_DEBUG@zoo_awget_children_@2874: Sending request xid=0x53801b14 for path [/aliases.json] to 127.0.0.1:2181
2014-05-24 08:07:53,471:20540(0x7f893209a700):ZOO_DEBUG@process_sync_completion@1870: Processing sync_completion with type=3 xid=0x53801b14 rc=0
8613 [0x7f8913fff700] TRACE zookeeper.adapter null - zk::ZooKeeperAdapter::getNodeChildren(std::vector<std::basic_string<char> >&, const string&, zk::ZKEventListener*, void*)::Trace::~Trace() 0x434ede Exit
8613 [0x7f8913fff700] DEBUG zkfuse null - update children done
8613 [0x7f8913fff700] DEBUG zkfuse null - node first use or reuse
8613 [0x7f8913fff700] DEBUG zkfuse null - update data
8613 [0x7f8913fff700] TRACE zookeeper.adapter null - zk::ZooKeeperAdapter::getNodeData(const string&, zk::ZKEventListener*, void*, Stat*)::Trace::Trace(const void*) 0x434e82 Enter
8613 [0x7f8913fff700] TRACE zookeeper.adapter null - zk::ZooKeeperAdapter::validatePath(const string&)::Trace::Trace(const void*) 0x434cc4 Enter
8613 [0x7f8913fff700] TRACE zookeeper.adapter null - zk::ZooKeeperAdapter::validatePath(const string&)::Trace::~Trace() 0x434cc4 Exit
8614 [0x7f8913fff700] TRACE zookeeper.adapter null - zk::ZooKeeperAdapter::verifyConnection()::Trace::Trace(const void*) 0x434cb3 Enter
8614 [0x7f8913fff700] TRACE zookeeper.adapter null - zk::ZooKeeperAdapter::verifyConnection()::Trace::~Trace() 0x434cb3 Exit
2014-05-24 08:07:53,472:20540(0x7f8913fff700):ZOO_DEBUG@zoo_awget@2661: Sending request xid=0x53801b15 for path [/aliases.json] to 127.0.0.1:2181
2014-05-24 08:07:53,472:20540(0x7f893209a700):ZOO_DEBUG@process_sync_completion@1870: Processing sync_completion with type=2 xid=0x53801b15 rc=0
8614 [0x7f8913fff700] TRACE zookeeper.adapter null - zk::ZooKeeperAdapter::getNodeData(const string&, zk::ZKEventListener*, void*, Stat*)::Trace::~Trace() 0x434e82 Exit
terminate called after throwing an instance of 'std::length_error'
what(): basic_string::_S_create
394689 No Perforce job exists for this issue. 1 394825
4 years, 17 weeks, 5 days ago Fix a bug in zkfuse that causes an abort upon reading a node's content 0|i1vyfr:
ZooKeeper ZOOKEEPER-1928

add configurable throttling to the number of snapshots concurrently sent by a leader

Improvement Resolved Major Fixed Edward Carter Edward Carter Edward Carter 21/May/14 20:46   10/Jun/14 07:16 09/Jun/14 18:19     server   0 6 604800 604800 0% We want to add configurable throttling to the number of snapshots concurrently sent by a leader. Without this, when recovering from a partial outage or network partition, the leader can become overloaded and unresponsive due to its attempts to send snapshots to too many followers and observers all at once. The throttle will operate by terminating the connection of any observer receiving a snapshot deemed to be in excess of the throttle. Followers should be allowed to receive snapshots unconditionally, though those snapshots do count against the quota.

I have a patch ready which implements this.
0% 0% 604800 604800 394186 No Perforce job exists for this issue. 3 394324
5 years, 41 weeks, 2 days ago 0|i1vvdz:
ZooKeeper ZOOKEEPER-1927

zkServer.sh fails to read dataDir (and others) from zoo.cfg on Solaris 10 (grep issue, manifests as FAILED TO WRITE PID).

Bug Closed Major Fixed Chris Nauroth Ed Schmed Ed Schmed 10/May/14 14:06   07/Sep/16 23:58 25/Aug/15 01:14 3.4.6 3.4.7, 3.5.3, 3.6.0 scripts   0 8   Solaris 5.10 Fails to write PID file with a permissions error, because the startup script fails to read the dataDir variable from zoo.cfg, and then tries to use the drive root ( / ) as the data dir.

Tracked the problem down to line 84 of zkServer.sh:

ZOO_DATADIR="$(grep "^[[:space:]]*dataDir" "$ZOOCFG" | sed -e 's/.*=//')"

If i run just that line and point it right at the config file, ZOO_DATADIR is empty.

If I remove [[:space:]]* from the grep:

ZOO_DATADIR="$(grep "^dataDir" "$ZOOCFG" | sed -e 's/.*=//')"

Then it works fine. (If I also make the same change on line 164 and 169)

My regex skills are pretty bad, so I'm afraid to comment on why [[space]]* needs to be in there?
391785 No Perforce job exists for this issue. 3 391988
3 years, 28 weeks ago 0|i1vh2v:
ZooKeeper ZOOKEEPER-1926

Unit tests should only use build/test/data for data

Bug Resolved Major Fixed Enis Soztutar Enis Soztutar Enis Soztutar 08/May/14 18:34   20/May/14 07:09 09/May/14 17:40   3.4.7, 3.5.0 tests   0 4   Some of the unit tests are creating temp files under system tmp dir (/tmp), and put data there.

We should encapsulate all temporary data from unit tests under build/test/data. ant clean will clean all data from previous runs.

391521 No Perforce job exists for this issue. 2 391734
5 years, 44 weeks, 2 days ago 0|i1vfin:
ZooKeeper ZOOKEEPER-1925

Time to Live or auto expiration of zookeeper node

New Feature Resolved Major Duplicate Unassigned Omkar Vinit Joshi Omkar Vinit Joshi 08/May/14 15:53   19/Dec/19 18:01 02/Dec/16 19:48         2 9   Today whenever a znode is created; it stays there for ever. We all know that there is a limitation in terms of how many nodes a system can handle. It would be nice to have a way to specify expiry time for every znode thereby stale zondes are cleandup automatically after sufficiently large time. any thoughts? ttl_nodes 391485 No Perforce job exists for this issue. 0 391699
4 years, 35 weeks, 3 days ago 0|i1vfav:
ZooKeeper ZOOKEEPER-1924

\,/ 'n;kj kkln. l,

Bug Resolved Major Invalid Unassigned Shwetha GS Shwetha GS 07/May/14 08:28   07/May/14 08:28 07/May/14 08:28         0 1   391136 No Perforce job exists for this issue. 0 391357
5 years, 46 weeks, 1 day ago 0|i1vd73:
ZooKeeper ZOOKEEPER-1923

A typo in zookeeperStarted document

Bug Resolved Minor Fixed Chengwei Yang Chengwei Yang Chengwei Yang 05/May/14 09:49   07/Aug/17 05:02 08/May/14 17:47 3.4.6 3.5.0 documentation   0 7   The trunk branch There is a typo in the document zookeeperStarted.*, see http://zookeeper.apache.org/doc/trunk/zookeeperStarted.html, in the section *Connecting to ZooKeeper*, where the *help* output *createpath* which should be *create path*. 390627 No Perforce job exists for this issue. 1 390873
2 years, 32 weeks, 3 days ago 0|i1va87:
ZooKeeper ZOOKEEPER-1922

Negative min latency value

Bug Open Minor Unresolved Unassigned J Potter J Potter 05/May/14 08:21   02/Jul/18 08:34   3.4.6   server   1 6   We're seeing the output of stat on one node return a negative value for min latency time:

stat
Zookeeper version: 3.4.6-1569965, built on 02/20/2014 09:09 GMT
Clients:
...
Latency min/avg/max: -477/149/261002

(The max value seems suspicious, too.)

Figured I'd report this, as I don't see any mention of it online or in other bug reports. Maybe negative values shouldn't be recorded?
390614 No Perforce job exists for this issue. 0 390860
1 year, 37 weeks, 3 days ago 0|i1va5b:
ZooKeeper ZOOKEEPER-1921

Zk binding to 127.0.0.1 for quorum connections

Bug Resolved Major Not A Problem Unassigned Nathan Neulinger Nathan Neulinger 04/May/14 11:28   23/Dec/14 15:57 23/Dec/14 15:57 3.4.6   quorum   0 3   I did not see this behavior with 3.4.5. When I upgraded to 3.4.6, couldn't establish quorum any more - what I found was that the listener on :3888 was only running on 127.0.0.1.

I was able to work around it by forcing dns to get the external hostnames, or removing 'my' hostname from /etc/hosts.

Key point - this appears to be a significant change in behavior from 3.4.5 - which I did not have any problems with...

I know you can specify the clientPortAddress - is there any way in the configuration to specify which address should be used for quorum connection listeners - or to force it to listen on 0.0.0.0 for quorum connections?
390512 No Perforce job exists for this issue. 0 390765
5 years, 13 weeks, 2 days ago 0|i1v9k7:
ZooKeeper ZOOKEEPER-1920

Login thread is not shutdown when close the ClientCnxn

Bug Open Minor Unresolved Rakesh Radhakrishnan liuyang liuyang 27/Apr/14 22:59   30/Apr/15 07:26   3.4.6   java client   0 7   A new ZooKeeper client will start three threads, the SendThread, EventThread and LoginThread. I belive that these threads will be shutdown after call the zk.close. I test that the SendThread and EventThread will be die, but LoginThread is still alive. The stack is:

"Thread-0" daemon prio=10 tid=0x00007ffcf0020000 nid=0x69c8 waiting on condition [0x00007ffd3cc25000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at org.apache.zookeeper.Login$1.run(Login.java:183)
at java.lang.Thread.run(Thread.java:744)
389164 No Perforce job exists for this issue. 0 389410
4 years, 47 weeks ago 0|i1v193:
ZooKeeper ZOOKEEPER-1919

Update the C implementation of removeWatches to have it match ZOOKEEPER-1910

Bug Closed Blocker Fixed Raúl Gutiérrez Segalés Raúl Gutiérrez Segalés Raúl Gutiérrez Segalés 24/Apr/14 22:44   20/May/19 13:50 29/May/18 19:29 3.5.0 3.6.0, 3.5.5 c client   0 5 0 7200   See https://issues.apache.org/jira/browse/ZOOKEEPER-1910 100% 100% 7200 0 pull-request-available, remove_watches 388802 No Perforce job exists for this issue. 2 389050
1 year, 42 weeks, 1 day ago 0|i1uz1j:
ZooKeeper ZOOKEEPER-1918

Add 64 bit Windows as a supported development platform

Task Resolved Minor Fixed Michi Mutsuzaki Michi Mutsuzaki Michi Mutsuzaki 24/Apr/14 16:04   03/Jul/14 07:23 02/Jul/14 23:57   3.5.0 documentation   0 5   Change this line:

Win32 is supported as a development platform only for both server and client.

to:

Win32 and Win64 are supported as a development platform only for both server and client.

http://zookeeper.apache.org/doc/trunk/zookeeperAdmin.html#sc_supportedPlatforms
388727 No Perforce job exists for this issue. 1 388977
5 years, 38 weeks ago
Reviewed
0|i1uylr:
ZooKeeper ZOOKEEPER-1917

Apache Zookeeper logs cleartext admin passwords

Bug Resolved Blocker Fixed Flavio Paiva Junqueira Flavio Paiva Junqueira Flavio Paiva Junqueira 22/Apr/14 14:02   13/Oct/14 00:22 12/Oct/14 23:16   3.4.7, 3.5.1, 3.6.0     0 6   Check the CVE entry for a description:

http://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2014-0085
388154 No Perforce job exists for this issue. 3 388411
5 years, 23 weeks, 3 days ago 0|i1uv4f:
ZooKeeper ZOOKEEPER-1916

make Cli depend on zookeeper

Improvement Patch Available Minor Unresolved Michi Mutsuzaki Michi Mutsuzaki Michi Mutsuzaki 19/Apr/14 19:56   14/Dec/19 06:07     3.7.0 c client   0 1   windows visual c++ 2008 Currently building the solution fails because the dependency is not properly set. 387749 No Perforce job exists for this issue. 1 388010
5 years, 48 weeks, 4 days ago 0|i1usnr:
ZooKeeper ZOOKEEPER-1915

Use $(ProjectDir) macro to specify include directories

Improvement In Progress Minor Unresolved Michi Mutsuzaki Michi Mutsuzaki Michi Mutsuzaki 19/Apr/14 19:55   05/Feb/20 07:11     3.7.0, 3.5.8 c client   0 1   windows visual c++ 2008 Right now we need to explicitly set the ZOOKEEPER_HOME environment variable. 387748 No Perforce job exists for this issue. 1 388009
5 years, 48 weeks, 5 days ago 0|i1usnj:
ZooKeeper ZOOKEEPER-1914

TestWatchers.cc failure

Bug Open Major Unresolved Michi Mutsuzaki Michi Mutsuzaki Michi Mutsuzaki 18/Apr/14 18:51   05/Feb/20 07:16     3.7.0, 3.5.8 c client   0 4   https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/2051/console

[exec] [exec] /home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/trunk/src/c/tests/TestWatchers.cc:667: Assertion: assertion failed [Expression: ensureCondition( deliveryTracker.deliveryCounterEquals(2),1000)<1000]
387686 No Perforce job exists for this issue. 2 387947
5 years, 35 weeks, 6 days ago 0|i1us9r:
ZooKeeper ZOOKEEPER-1913

Invalid manifest files due to bogus revision property value

Bug Resolved Major Fixed Raúl Gutiérrez Segalés Raúl Gutiérrez Segalés Raúl Gutiérrez Segalés 18/Apr/14 16:42   19/Apr/14 07:06 18/Apr/14 18:35 3.4.6, 3.5.0 3.4.7, 3.5.0 build   0 6   Without the proposed patch, I get invalid manifests because stderr is added to the revision property. I think this might be something specific to my setup though:

{noformat}
$ java -version
Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=utf8
java version "1.7.0_51"
OpenJDK Runtime Environment (fedora-2.4.5.1.fc20-x86_64 u51-b31)
OpenJDK 64-Bit Server VM (build 24.51-b03, mixed mode)
{noformat}

since it doesn't seem happen with older java/ant combinations.

Nonetheless, it seems like the right thing is to explicitly ignore stderr.
387663 No Perforce job exists for this issue. 1 387925
5 years, 48 weeks, 5 days ago 0|i1us47:
ZooKeeper ZOOKEEPER-1912

Leader election lets 2 leaders happen

Bug Resolved Critical Not A Problem Flavio Paiva Junqueira Tanakorn Leesatapornwongsa Tanakorn Leesatapornwongsa 14/Apr/14 16:14   06/Feb/16 23:13 06/Feb/16 23:13 3.4.6   leaderElection   0 4   Ubuntu 12.04, OpenJDK 1.6 In 3-node cluster, when there are 2 nodes die and reboot during leader election, it might lead to the case that there are 2 leaders happen in the system. Eventually, a leader that does not has follower supports and quit being leader, but it makes us lose some availability.

I am building a tools that can reorder messages and disk write, and also inject node crash to the system and found this bug.
These are the step of events that my tools execute in sequence that lead to 2 leaders at the end.
My zookeeper nodes have id = 0,1,2

packetsend from=0 to=1 state=0 leader=0 zxid=0 electionEpoch=1 peerEpoch=0
packetsend from=0 to=2 state=0 leader=0 zxid=0 electionEpoch=1 peerEpoch=0
packetsend from=2 to=0 state=0 leader=2 zxid=0 electionEpoch=1 peerEpoch=0
packetsend from=2 to=1 state=0 leader=2 zxid=0 electionEpoch=1 peerEpoch=0
packetsend from=1 to=0 state=0 leader=1 zxid=0 electionEpoch=1 peerEpoch=0
packetsend from=1 to=2 state=0 leader=1 zxid=0 electionEpoch=1 peerEpoch=0
packetsend from=1 to=0 state=0 leader=2 zxid=0 electionEpoch=1 peerEpoch=0
packetsend from=0 to=1 state=0 leader=2 zxid=0 electionEpoch=1 peerEpoch=0
packetsend from=1 to=2 state=0 leader=2 zxid=0 electionEpoch=1 peerEpoch=0
packetsend from=0 to=2 state=0 leader=2 zxid=0 electionEpoch=1 peerEpoch=0
diskwrite nodeId=0 write=currentEpoch
nodecrash id=0
nodecrash id=1
nodestart id=0
nodestart id=1
diskwrite nodeId=2 write=currentEpoch
packetsend from=2 to=0 state=0 leader=2 zxid=0 electionEpoch=1 peerEpoch=0
packetsend from=0 to=2 state=0 leader=0 zxid=0 electionEpoch=1 peerEpoch=1
packetsend from=2 to=1 state=0 leader=2 zxid=0 electionEpoch=1 peerEpoch=0
packetsend from=0 to=1 state=0 leader=0 zxid=0 electionEpoch=1 peerEpoch=1
packetsend from=1 to=0 state=0 leader=1 zxid=0 electionEpoch=1 peerEpoch=0
packetsend from=1 to=2 state=0 leader=1 zxid=0 electionEpoch=1 peerEpoch=0
packetsend from=2 to=0 state=2 leader=2 zxid=0 electionEpoch=1 peerEpoch=1
packetsend from=1 to=0 state=0 leader=0 zxid=0 electionEpoch=1 peerEpoch=1
packetsend from=2 to=1 state=2 leader=2 zxid=0 electionEpoch=1 peerEpoch=1
packetsend from=1 to=2 state=0 leader=0 zxid=0 electionEpoch=1 peerEpoch=1
packetsend from=2 to=1 state=2 leader=2 zxid=0 electionEpoch=1 peerEpoch=1
packetsend from=1 to=0 state=0 leader=0 zxid=0 electionEpoch=1 peerEpoch=1
packetsend from=1 to=2 state=0 leader=0 zxid=0 electionEpoch=1 peerEpoch=1
packetsend from=0 to=1 state=2 leader=0 zxid=0 electionEpoch=1 peerEpoch=1
packetsend from=2 to=1 state=2 leader=2 zxid=0 electionEpoch=1 peerEpoch=1
packetsend from=1 to=0 state=0 leader=0 zxid=0 electionEpoch=1 peerEpoch=1
packetsend from=1 to=2 state=0 leader=0 zxid=0 electionEpoch=1 peerEpoch=1
packetsend from=0 to=1 state=2 leader=0 zxid=0 electionEpoch=1 peerEpoch=1
packetsend from=2 to=1 state=2 leader=2 zxid=0 electionEpoch=1 peerEpoch=1
packetsend from=1 to=0 state=0 leader=0 zxid=0 electionEpoch=1 peerEpoch=1
packetsend from=1 to=2 state=0 leader=0 zxid=0 electionEpoch=1 peerEpoch=1
packetsend from=0 to=1 state=2 leader=0 zxid=0 electionEpoch=1 peerEpoch=1
packetsend from=2 to=0 state=0 leader=2 zxid=0 electionEpoch=2 peerEpoch=1
packetsend from=2 to=1 state=0 leader=2 zxid=0 electionEpoch=2 peerEpoch=1
packetsend from=0 to=2 state=2 leader=0 zxid=0 electionEpoch=1 peerEpoch=1
packetsend from=2 to=0 state=0 leader=2 zxid=0 electionEpoch=2 peerEpoch=1
packetsend from=1 to=0 state=0 leader=2 zxid=0 electionEpoch=2 peerEpoch=1
packetsend from=1 to=2 state=0 leader=2 zxid=0 electionEpoch=2 peerEpoch=1
packetsend from=2 to=1 state=0 leader=2 zxid=0 electionEpoch=2 peerEpoch=1
packetsend from=0 to=2 state=2 leader=0 zxid=0 electionEpoch=1 peerEpoch=1
packetsend from=2 to=0 state=0 leader=2 zxid=0 electionEpoch=2 peerEpoch=1
packetsend from=0 to=1 state=2 leader=0 zxid=0 electionEpoch=1 peerEpoch=1
packetsend from=2 to=1 state=0 leader=2 zxid=0 electionEpoch=2 peerEpoch=1
packetsend from=0 to=2 state=2 leader=0 zxid=0 electionEpoch=1 peerEpoch=1
diskwrite nodeId=2 write=currentEpoch
diskwrite nodeId=1 write=currentEpoch
386687 No Perforce job exists for this issue. 2 386951
4 years, 6 weeks, 5 days ago 0|i1um47:
ZooKeeper ZOOKEEPER-1911

REST contrib module does not include all required files when packaged

Bug Resolved Major Fixed Sean Mackrory Sean Mackrory Sean Mackrory 11/Apr/14 11:08   25/Apr/14 07:49 25/Apr/14 04:12 3.4.6, 3.5.0 3.4.7, 3.5.0     0 5   If you compile the REST contrib module, the tarball will only include the main JAR. It will not bundle the required dependencies or include the minimal working configuration files. 386287 No Perforce job exists for this issue. 2 386552
5 years, 47 weeks, 6 days ago 0|i1ujnj:
ZooKeeper ZOOKEEPER-1910

RemoveWatches wrongly removes the watcher if multiple watches exists on a path

Bug Resolved Major Fixed Rakesh Radhakrishnan Rakesh Radhakrishnan Rakesh Radhakrishnan 11/Apr/14 04:08   16/Dec/18 09:46 28/Apr/14 22:31   3.5.0 java client, server   0 7   Consider a case where zkclient has added 2 data watchers(say 'w1' and 'w2') on '/node1'.

Now user has removed w1, but this is deleting the 'CnxnWatcher' in ZK server against the "/node1" path. This will affect other data watchers(if any) of same client on same path. In our case 'w2' would not be notified.

Note: please see the attached test case to understand more.
remove_watches 386218 No Perforce job exists for this issue. 4 386483
1 year, 43 weeks, 1 day ago 0|i1uj87:
ZooKeeper ZOOKEEPER-1909

removeWatches doesn't return NOWATCHER when there is no watch set

Bug Resolved Major Fixed Raúl Gutiérrez Segalés Raúl Gutiérrez Segalés Raúl Gutiérrez Segalés 09/Apr/14 19:51   16/Dec/18 09:47 17/Apr/14 02:52 3.5.0 3.5.0 server   0 5   ZOOKEEPER-442 introduced support for a new opcode: removeWatches. The way it was implemented though, implies that you need to check on the client side if a watch/watcher is set *before* you send your request to the server. If you don't, ZK will just swallow your request and won't return an error code if there isn't a watch set for that path.

I noticed this whilst implementing removeWatches for Kazoo [1]. As mentioned, I guess it could be expected that clients should do the check on their side but I think that the correct thing would to have the server do the validation and return the error code accordingly as well.

[~rakeshr], [~phunt]: thoughts?

[1] https://github.com/rgs1/kazoo/commit/44ca48e975aeea3fd0664fe13136a72caf89e54f
remove_watches 385925 No Perforce job exists for this issue. 4 386189
5 years, 49 weeks ago 0|i1uhfb:
ZooKeeper ZOOKEEPER-1908

setAcl should be have a recursive function

Improvement Closed Major Fixed Reid Chan Kevin Odell Kevin Odell 01/Apr/14 18:09   04/Oct/19 10:55 12/Oct/18 11:08 3.4.6, 4.0.0 3.6.0, 3.5.5 scripts, server   2 8 0 22800   setAcl should be have a recursive function. This becomes a problem with HBase when trying to back in and out of secure clusters. 100% 100% 22800 0 pull-request-available 384374 No Perforce job exists for this issue. 0 384642
1 year, 22 weeks, 6 days ago 0|i1u7wn:
ZooKeeper ZOOKEEPER-1907

Improve Thread handling

Improvement Resolved Major Fixed Rakesh Radhakrishnan Rakesh Radhakrishnan Rakesh Radhakrishnan 31/Mar/14 11:18   06/Dec/16 22:03 17/Dec/15 04:13 3.5.0 3.4.7, 3.5.1, 3.6.0 server   0 8   Server has many critical threads running and co-ordinating each other like RequestProcessor chains et. When going through each threads, most of them having the similar structure like:
{code}
public void run() {
try {
while(running)
// processing logic
}
} catch (InterruptedException e) {
LOG.error("Unexpected interruption", e);
} catch (Exception e) {
LOG.error("Unexpected exception", e);
}
LOG.info("...exited loop!");
}
{code}

From the design I could see, there could be a chance of silently leaving the thread by swallowing the exception. If this happens in the production, the server would get hanged forever and would not be able to deliver its role. Now its hard for the management tool to detect this.

The idea of this JIRA is to discuss and imprv.

Reference: [Community discussion thread|http://mail-archives.apache.org/mod_mbox/zookeeper-user/201403.mbox/%3CC2496325850AA74C92AAF83AA9662D26458A1D67@szxeml561-mbx.china.huawei.com%3E]
384013 No Perforce job exists for this issue. 14 384281
3 years, 15 weeks, 1 day ago 0|i1u5on:
ZooKeeper ZOOKEEPER-1906

zkpython: invalid data in GetData for empty node

Bug Resolved Major Fixed Nikita Vetoshkin Nikita Vetoshkin Nikita Vetoshkin 31/Mar/14 03:52   02/Apr/14 07:10 01/Apr/14 17:02 3.4.6, 3.5.0 3.4.7, 3.5.0 contrib-bindings   0 4   FreeBSD In python if we ask {{zookeeper.get}} (which translates into {{pyzoo_get}}) for empty node we can get trash in result on Python level. Issue is pretty tricky. It goes like this:
* python C extension allocates buffer with malloc {{buffer = malloc(sizeof(char)*buffer_len);}} and calls {{zoo_wget}} providing both {{buffer}} and {{buffer_len}}.
* deserialize_GetDataResponse deserializes empty buffer and sets {{buffer_len}} to -1 and {{zoo_wget}} returns.
* python C extension calls {{Py_BuildValue( "(s#,N)", buffer,buffer_len ...}} with {{buffer_len}} set to -1.
* {{Py_BuildValue}} calls {{do_mkvalue}} to build python string which falls back to {{strlen(str)}} in case string length ({{buffer_len < 0}}) - that's our case.
* *usually* strlen returns 0, because e.g. linux uses magic zero filled page as result of mmap (which is being copied upon page fault, i.e. when you want to write to it)
* everything works!

But on FreeBSD (not always) we can get random data in {{malloc}} result and this trash will be exposed to the user.

Not sure about the right way to fix this, but something like
{noformat}
Index: src/contrib/zkpython/src/c/zookeeper.c
===================================================================
--- src/contrib/zkpython/src/c/zookeeper.c (revision 1583238)
+++ src/contrib/zkpython/src/c/zookeeper.c (working copy)
@@ -1223,7 +1223,7 @@
}

PyObject *stat_dict = build_stat( &stat );
- PyObject *ret = Py_BuildValue( "(s#,N)", buffer,buffer_len, stat_dict );
+ PyObject *ret = Py_BuildValue( "(s#,N)", buffer,buffer_len < 0 ? 0 : buffer_len, stat_dict );
free(buffer);

return ret;
{noformat}
should do the trick
383922 No Perforce job exists for this issue. 1 384190
5 years, 51 weeks, 1 day ago 0|i1u54f:
ZooKeeper ZOOKEEPER-1905

ZOOKEEPER-1833 ZKClients are hitting KeeperException$ConnectionLossException due to wrong usage pattern

Sub-task Resolved Major Won't Fix Rakesh Radhakrishnan Rakesh Radhakrishnan Rakesh Radhakrishnan 25/Mar/14 09:33   27/Mar/14 09:53 27/Mar/14 09:53   3.4.7 tests   0 2   Since the ZooKeeper client connection establishment happens in async way, the client should wait for the 'KeeperState.SyncConnected' event before start performing any ops. Many tests are having this kind of wrong pattern.

Reference:- Below stack trace taken from build https://builds.apache.org/job/ZooKeeper-3.4-WinVS2008_java/465/

{code}
[junit] 2014-03-19 08:36:53,056 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@62] - TEST METHOD FAILED testChecksums
[junit] org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /crctest- 942
[junit] at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
[junit] at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit] at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
[junit] at org.apache.zookeeper.server.CRCTest.testChecksums(CRCTest.java:127)
{code}
381797 No Perforce job exists for this issue. 0 382072
6 years ago 0|i1ts2f:
ZooKeeper ZOOKEEPER-1904

ZOOKEEPER-1833 WatcherTest#testWatchAutoResetWithPending is failing

Sub-task Resolved Major Fixed Rakesh Radhakrishnan Rakesh Radhakrishnan Rakesh Radhakrishnan 25/Mar/14 04:05   27/Mar/14 07:10 26/Mar/14 17:17   3.4.7, 3.5.0 tests   0 4   Following is the stacktrace taken from [Build : ZooKeeper-3.4-WinVS2008_java/465|https://builds.apache.org/job/ZooKeeper-3.4-WinVS2008_java/465/]

{code}
[junit] 2014-03-19 09:28:50,020 [myid:] - INFO [main-SendThread(127.0.0.1:11278):ClientCnxn$SendThread@975] - Opening socket connection to server 127.0.0.1/127.0.0.1:11278. Will not attempt to authenticate using SASL (unknown error)
[junit] 2014-03-19 09:28:51,025 [myid:] - WARN [main-SendThread(127.0.0.1:11278):ClientCnxn$SendThread@1102] - Session 0x144d9ab1f9e0000 for server null, unexpected error, closing socket connection and attempting reconnect
[junit] java.net.ConnectException: Connection refused: no further information
[junit] at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
[junit] at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:701)
[junit] at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
[junit] at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
[junit] 2014-03-19 09:28:52,661 [myid:] - INFO [main-SendThread(127.0.0.1:11278):ClientCnxn$SendThread@975] - Opening socket connection to server 127.0.0.1/127.0.0.1:11278. Will not attempt to authenticate using SASL (unknown error)
[junit] 2014-03-19 09:28:53,640 [myid:] - WARN [main-SendThread(127.0.0.1:11278):ClientCnxn$SendThread@1102] - Session 0x144d9ab1f9e0000 for server null, unexpected error, closing socket connection and attempting reconnect
[junit] java.net.ConnectException: Connection refused: no further information
[junit] at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
[junit] at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:701)
[junit] at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
[junit] at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
[junit] 2014-03-19 09:28:55,435 [myid:] - INFO [main-SendThread(127.0.0.1:11278):ClientCnxn$SendThread@975] - Opening socket connection to server 127.0.0.1/127.0.0.1:11278. Will not attempt to authenticate using SASL (unknown error)
[junit] 2014-03-19 09:28:56,111 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@62] - TEST METHOD FAILED testWatchAutoResetWithPending
[junit] java.util.concurrent.TimeoutException: Did not disconnect
[junit] at org.apache.zookeeper.test.ClientBase$CountdownWatcher.waitForDisconnected(ClientBase.java:145)
[junit] at org.apache.zookeeper.test.WatcherTest.testWatchAutoResetWithPending(WatcherTest.java:202)
[junit] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
{code}
381756 No Perforce job exists for this issue. 3 382031
6 years ago 0|i1trtb:
ZooKeeper ZOOKEEPER-1903

30/6

Bug Resolved Major Invalid Unassigned ROY Assink ROY Assink 24/Mar/14 10:52   15/Apr/14 13:59 15/Apr/14 13:59         0 2   381546 No Perforce job exists for this issue. 0 381821
5 years, 49 weeks, 2 days ago 0|i1tqiv:
ZooKeeper ZOOKEEPER-1902

summertime

Bug Resolved Major Invalid Unassigned ROY Assink ROY Assink 24/Mar/14 10:51   24/Mar/14 14:11 24/Mar/14 14:11         0 3   381545 No Perforce job exists for this issue. 0 381820
6 years, 3 days ago 0|i1tqin:
ZooKeeper ZOOKEEPER-1901

[JDK8] Sort children for comparison in AsyncOps tests

Bug Resolved Minor Fixed Andrew Kyle Purtell Andrew Kyle Purtell Andrew Kyle Purtell 23/Mar/14 16:08   25/Mar/14 12:30 23/Mar/14 17:04 3.4.6 3.4.7, 3.5.0 tests   0 4   AsyncOpsTest, ChrootAsyncTest, and NioNettySuiteTest can fail running on Java 8 if child znodes are not added to a list in the same order as expected.

For example

{noformat}
Testcase: testAsyncGetChildrenTwo took 0.166 sec
FAILED
expected:<OK:/foo:[child[1, child2]]> but was:<OK:/foo:[child[2, child1]]>
junit.framework.AssertionFailedError: expected:<OK:/foo:[child[1, child2]]> but was:<OK:/foo:[child[2, child1]]>
at org.apache.zookeeper.test.AsyncOps$AsyncCB.verify(AsyncOps.java:113)
at org.apache.zookeeper.test.AsyncOps$ChildrenCB.verify(AsyncOps.java:298)
at org.apache.zookeeper.test.AsyncOps$ChildrenCB.verifyGetChildrenTwo(AsyncOps.java:287)
at org.apache.zookeeper.test.AsyncOpsTest.testAsyncGetChildrenTwo(AsyncOpsTest.java:155)
at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:52)
{noformat}

{noformat}
Testcase: testAsyncGetChildren2Two took 0.154 sec
FAILED
expected:<OK:/foo:[child[1, child2]]> but was:<OK:/foo:[child[2, child1]]>
junit.framework.AssertionFailedError: expected:<OK:/foo:[child[1, child2]]> but was:<OK:/foo:[child[2, child1]]>
at org.apache.zookeeper.test.AsyncOps$AsyncCB.verify(AsyncOps.java:113)
at org.apache.zookeeper.test.AsyncOps$Children2CB.verify(AsyncOps.java:383)
at org.apache.zookeeper.test.AsyncOps$Children2CB.verifyGetChildrenTwo(AsyncOps.java:372)
at org.apache.zookeeper.test.AsyncOpsTest.testAsyncGetChildren2Two(AsyncOpsTest.java:175)
at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:52)
{noformat}

This seems like a test only issue because getChildren javadoc says "The list of children returned is not sorted and no guarantee is provided as to its natural or lexical order." So, fix the tests by sorting the incoming lists.
381434 No Perforce job exists for this issue. 2 381711
6 years, 3 days ago 0|i1tpuf:
ZooKeeper ZOOKEEPER-1900

NullPointerException in truncate

Bug Resolved Blocker Fixed Camille Fournier Steven Bower Steven Bower 21/Mar/14 14:11   01/Jul/14 07:11 30/Jun/14 13:20 3.4.5, 3.4.6 3.4.7, 3.5.0     0 8   linux java 1.6 The other day we started up a ZK instance that had been down for a bit (1day) and started getting NPEs all over the place...

{noformat}
2014-20-03 11:15:42.320 INFO QuorumPeerConfig [main] - Reading configuration from: /xxx/bin/zk/etc/zk.cfg
2014-20-03 11:15:42.350 INFO QuorumPeerConfig [main] - Defaulting to majority quorums
2014-20-03 11:15:42.353 INFO DatadirCleanupManager [main] - autopurge.snapRetainCount set to 3
2014-20-03 11:15:42.353 INFO DatadirCleanupManager [main] - autopurge.purgeInterval set to 0
2014-20-03 11:15:42.353 INFO DatadirCleanupManager [main] - Purge task is not scheduled.
2014-20-03 11:15:42.385 INFO QuorumPeerMain [main] - Starting quorum peer
2014-20-03 11:15:42.399 INFO NIOServerCnxnFactory [main] - binding to port 0.0.0.0/0.0.0.0:5555
2014-20-03 11:15:42.413 INFO QuorumPeer [main] - tickTime set to 2000
2014-20-03 11:15:42.413 INFO QuorumPeer [main] - minSessionTimeout set to -1
2014-20-03 11:15:42.413 INFO QuorumPeer [main] - maxSessionTimeout set to -1
2014-20-03 11:15:42.413 INFO QuorumPeer [main] - initLimit set to 10
2014-20-03 11:15:42.456 INFO FileSnap [main] - Reading snapshot /xxx/zk_data/version-2/snapshot.2c00000000
2014-20-03 11:15:42.463 INFO QuorumCnxManager [Thread-3] - My election bind port: 0.0.0.0/0.0.0.0:7555
2014-20-03 11:15:42.470 INFO QuorumPeer [QuorumPeer[myid=3]/0:0:0:0:0:0:0:0:5555] - LOOKING
2014-20-03 11:15:42.471 INFO FastLeaderElection [QuorumPeer[myid=3]/0:0:0:0:0:0:0:0:5555] - New election. My id = 3, proposed zxid=0x8000000000000000
2014-20-03 11:15:42.479 INFO FastLeaderElection [WorkerReceiver[myid=3]] - Notification: 2 (n.leader), 0x2b00000002 (n.zxid), 0x2c (n.round), FOLLOWING (n.state), 1 (n.sid), 0x2b (n.peerEPoch), LOOKING (my state)
2014-20-03 11:15:42.479 INFO FastLeaderElection [WorkerReceiver[myid=3]] - Notification: 2 (n.leader), 0x2b00000002 (n.zxid), 0x2c (n.round), FOLLOWING (n.state), 1 (n.sid), 0x2b (n.peerEPoch), LOOKING (my state)
2014-20-03 11:15:42.482 INFO QuorumCnxManager [WorkerSender[myid=3]] - Have smaller server identifier, so dropping the connection: (5, 3)
2014-20-03 11:15:42.482 INFO FastLeaderElection [WorkerReceiver[myid=3]] - Notification: 2 (n.leader), 0x2b00000002 (n.zxid), 0x2c (n.round), LEADING (n.state), 2 (n.sid), 0x2b (n.peerEPoch), LOOKING (my state)
2014-20-03 11:15:42.482 INFO FastLeaderElection [WorkerReceiver[myid=3]] - Notification: 2 (n.leader), 0x2b00000002 (n.zxid), 0x2c (n.round), LEADING (n.state), 2 (n.sid), 0x2b (n.peerEPoch), LOOKING (my state)
2014-20-03 11:15:42.482 INFO QuorumPeer [QuorumPeer[myid=3]/0:0:0:0:0:0:0:0:5555] - OBSERVING
2014-20-03 11:15:42.486 INFO Learner [QuorumPeer[myid=3]/0:0:0:0:0:0:0:0:5555] - TCP NoDelay set to: true
2014-20-03 11:15:42.488 INFO QuorumCnxManager [host1/###.###.###.###:7555] - Received connection request /###.###.###.###:64528
2014-20-03 11:15:42.490 INFO ZooKeeperServer [QuorumPeer[myid=3]/0:0:0:0:0:0:0:0:5555] - Server environment:zookeeper.version=3.4.5-1392090, built on 09/30/2012 17:52 GMT
2014-20-03 11:15:42.490 INFO ZooKeeperServer [QuorumPeer[myid=3]/0:0:0:0:0:0:0:0:5555] - Server environment:host.name=host1
2014-20-03 11:15:42.490 INFO ZooKeeperServer [QuorumPeer[myid=3]/0:0:0:0:0:0:0:0:5555] - Server environment:java.version=1.6.0_20
2014-20-03 11:15:42.490 INFO ZooKeeperServer [QuorumPeer[myid=3]/0:0:0:0:0:0:0:0:5555] - Server environment:java.vendor=Sun Microsystems Inc.
2014-20-03 11:15:42.490 INFO ZooKeeperServer [QuorumPeer[myid=3]/0:0:0:0:0:0:0:0:5555] - Server environment:java.home=/xxx/util/common/jdk1.6.0_20_64bit/jre
2014-20-03 11:15:42.490 INFO ZooKeeperServer [QuorumPeer[myid=3]/0:0:0:0:0:0:0:0:5555] - Server environment:java.class.path=/xxx/bin/zk/etc:/xxx/bin/zk/lib/slf4j-log4j12-1.7.2.jar:/xxx/bin/zk/lib/jline-0.9.94.jar:/xxx/bin/zk/lib/jul-to-slf4j-1.7.2.jar:/xxx/bin/zk/lib/ZooInspector-3.4.5.jar:/xxx/bin/zk/lib/jcl-over-slf4j-1.7.2.jar:/xxx/bin/zk/lib/log4j-1.2.17.jar:/xxx/bin/zk/lib/zookeeper-3.4.5.jar:/xxx/bin/zk/lib/slf4j-api-1.7.2.jar:/xxx/bin/zk/lib/netty-3.2.2.Final.jar
2014-20-03 11:15:42.490 INFO ZooKeeperServer [QuorumPeer[myid=3]/0:0:0:0:0:0:0:0:5555] - Server environment:java.library.path=/xxx/util/common/jdk1.6.0_20_64bit/jre/lib/amd64/server:/xxx/util/common/jdk1.6.0_20_64bit/jre/lib/amd64:/xxx/util/common/jdk1.6.0_20_64bit/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2014-20-03 11:15:42.490 INFO ZooKeeperServer [QuorumPeer[myid=3]/0:0:0:0:0:0:0:0:5555] - Server environment:java.io.tmpdir=/tmp
2014-20-03 11:15:42.490 INFO ZooKeeperServer [QuorumPeer[myid=3]/0:0:0:0:0:0:0:0:5555] - Server environment:java.compiler=<NA>
2014-20-03 11:15:42.490 INFO ZooKeeperServer [QuorumPeer[myid=3]/0:0:0:0:0:0:0:0:5555] - Server environment:os.name=Linux
2014-20-03 11:15:42.490 INFO ZooKeeperServer [QuorumPeer[myid=3]/0:0:0:0:0:0:0:0:5555] - Server environment:os.arch=amd64
2014-20-03 11:15:42.490 INFO ZooKeeperServer [QuorumPeer[myid=3]/0:0:0:0:0:0:0:0:5555] - Server environment:os.version=2.6.32-220.2.1.el6.x86_64
2014-20-03 11:15:42.490 INFO ZooKeeperServer [QuorumPeer[myid=3]/0:0:0:0:0:0:0:0:5555] - Server environment:user.name=op
2014-20-03 11:15:42.490 INFO ZooKeeperServer [QuorumPeer[myid=3]/0:0:0:0:0:0:0:0:5555] - Server environment:user.home=/xxx/bin
2014-20-03 11:15:42.490 INFO ZooKeeperServer [QuorumPeer[myid=3]/0:0:0:0:0:0:0:0:5555] - Server environment:user.dir=/xxx/bin
2014-20-03 11:15:42.491 INFO ZooKeeperServer [QuorumPeer[myid=3]/0:0:0:0:0:0:0:0:5555] - Created server with tickTime 2000 minSessionTimeout 4000 maxSessionTimeout 40000 datadir /xxx/zk_log/version-2 snapdir /xxx/zk_data/version-2
2014-20-03 11:15:42.493 INFO Learner [QuorumPeer[myid=3]/0:0:0:0:0:0:0:0:5555] - Observing host4/###.###.###.###:6555
2014-20-03 11:15:42.495 INFO FastLeaderElection [WorkerReceiver[myid=3]] - Notification: 2 (n.leader), 0x2b00000002 (n.zxid), 0x2c (n.round), FOLLOWING (n.state), 5 (n.sid), 0x2b (n.peerEPoch), OBSERVING (my state)
2014-20-03 11:15:42.498 WARN Learner [QuorumPeer[myid=3]/0:0:0:0:0:0:0:0:5555] - Truncating log to get in sync with the leader 0x2b00000002
2014-20-03 11:15:42.499 WARN QuorumPeer [QuorumPeer[myid=3]/0:0:0:0:0:0:0:0:5555] - Unexpected exception
java.lang.NullPointerException
at org.apache.zookeeper.server.persistence.FileTxnLog.truncate(FileTxnLog.java:352)
at org.apache.zookeeper.server.persistence.FileTxnSnapLog.truncateLog(FileTxnSnapLog.java:259)
at org.apache.zookeeper.server.ZKDatabase.truncateLog(ZKDatabase.java:438)
at org.apache.zookeeper.server.quorum.Learner.syncWithLeader(Learner.java:339)
at org.apache.zookeeper.server.quorum.Observer.observeLeader(Observer.java:72)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:727)
2014-20-03 11:15:42.500 INFO Learner [QuorumPeer[myid=3]/0:0:0:0:0:0:0:0:5555] - shutdown called
java.lang.Exception: shutdown Observer
at org.apache.zookeeper.server.quorum.Observer.shutdown(Observer.java:137)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:731)
2014-20-03 11:15:42.500 INFO ZooKeeperServer [QuorumPeer[myid=3]/0:0:0:0:0:0:0:0:5555] - shutting down
2014-20-03 11:15:42.500 INFO QuorumPeer [QuorumPeer[myid=3]/0:0:0:0:0:0:0:0:5555] - LOOKING
2014-20-03 11:15:42.501 INFO FastLeaderElection [QuorumPeer[myid=3]/0:0:0:0:0:0:0:0:5555] - New election. My id = 3, proposed zxid=0x8000000000000000
2014-20-03 11:15:42.503 INFO FastLeaderElection [WorkerReceiver[myid=3]] - Notification: 2 (n.leader), 0x2b00000002 (n.zxid), 0x2c (n.round), FOLLOWING (n.state), 1 (n.sid), 0x2b (n.peerEPoch), LOOKING (my state)
2014-20-03 11:15:42.503 INFO FastLeaderElection [WorkerReceiver[myid=3]] - Notification: 2 (n.leader), 0x2b00000002 (n.zxid), 0x2c (n.round), LEADING (n.state), 2 (n.sid), 0x2b (n.peerEPoch), LOOKING (my state)
2014-20-03 11:15:42.503 INFO QuorumPeer [QuorumPeer[myid=3]/0:0:0:0:0:0:0:0:5555] - OBSERVING
2014-20-03 11:15:42.503 INFO ZooKeeperServer [QuorumPeer[myid=3]/0:0:0:0:0:0:0:0:5555] - Created server with tickTime 2000 minSessionTimeout 4000 maxSessionTimeout 40000 datadir /xxx/zk_log/version-2 snapdir /xxx/zk_data/version-2
2014-20-03 11:15:42.504 INFO Learner [QuorumPeer[myid=3]/0:0:0:0:0:0:0:0:5555] - Observing host4/###.###.###.###:6555
2014-20-03 11:15:42.504 INFO FastLeaderElection [WorkerReceiver[myid=3]] - Notification: 2 (n.leader), 0x2b00000002 (n.zxid), 0x2c (n.round), FOLLOWING (n.state), 5 (n.sid), 0x2b (n.peerEPoch), OBSERVING (my state)
2014-20-03 11:15:42.514 INFO FileSnap [QuorumPeer[myid=3]/0:0:0:0:0:0:0:0:5555] - Reading snapshot /xxx/zk_data/version-2/snapshot.2c00000000
2014-20-03 11:15:42.517 WARN Learner [QuorumPeer[myid=3]/0:0:0:0:0:0:0:0:5555] - Truncating log to get in sync with the leader 0x2b00000002
2014-20-03 11:15:42.518 WARN QuorumPeer [QuorumPeer[myid=3]/0:0:0:0:0:0:0:0:5555] - Unexpected exception
java.lang.NullPointerException
at org.apache.zookeeper.server.persistence.FileTxnLog.truncate(FileTxnLog.java:352)
at org.apache.zookeeper.server.persistence.FileTxnSnapLog.truncateLog(FileTxnSnapLog.java:259)
at org.apache.zookeeper.server.ZKDatabase.truncateLog(ZKDatabase.java:438)
at org.apache.zookeeper.server.quorum.Learner.syncWithLeader(Learner.java:339)
at org.apache.zookeeper.server.quorum.Observer.observeLeader(Observer.java:72)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:727)
{noformat}

This exception went on and on over and over again (more than 1M times in a day) until it then began spewing this exception:

{noformat}
2014-20-03 13:45:32.843 INFO QuorumPeer [QuorumPeer[myid=3]/0:0:0:0:0:0:0:0:5555] - LOOKING
2014-20-03 13:45:32.843 INFO FastLeaderElection [QuorumPeer[myid=3]/0:0:0:0:0:0:0:0:5555] - New election. My id = 3, proposed zxid=0x8000000000000000
2014-20-03 13:45:32.844 INFO FastLeaderElection [WorkerReceiver[myid=3]] - Notification: 2 (n.leader), 0x2b00000002 (n.zxid), 0x2c (n.round), FOLLOWING (n.state), 1 (n.sid), 0x2b (n.peerEPoch), LOOKING (my state)
2014-20-03 13:45:32.845 INFO FastLeaderElection [WorkerReceiver[myid=3]] - Notification: 2 (n.leader), 0x2b00000002 (n.zxid), 0x2c (n.round), LEADING (n.state), 2 (n.sid), 0x2b (n.peerEPoch), LOOKING (my state)
2014-20-03 13:45:32.845 INFO QuorumPeer [QuorumPeer[myid=3]/0:0:0:0:0:0:0:0:5555] - OBSERVING
2014-20-03 13:45:32.845 INFO FastLeaderElection [WorkerReceiver[myid=3]] - Notification: 2 (n.leader), 0x2b00000002 (n.zxid), 0x2c (n.round), FOLLOWING (n.state), 5 (n.sid), 0x2b (n.peerEPoch), OBSERVING (my state)
2014-20-03 13:45:32.845 INFO ZooKeeperServer [QuorumPeer[myid=3]/0:0:0:0:0:0:0:0:5555] - Created server with tickTime 2000 minSessionTimeout 4000 maxSessionTimeout 40000 datadir /xxx/zk_log/version-2 snapdir /xxx/zk_data/version-2
2014-20-03 13:45:32.845 INFO Learner [QuorumPeer[myid=3]/0:0:0:0:0:0:0:0:5555] - Observing host4/###.###.###.###:6555
2014-20-03 13:45:32.853 WARN Learner [QuorumPeer[myid=3]/0:0:0:0:0:0:0:0:5555] - Unexpected exception, tries=0, connecting to host4/###.###.###.###:6555
java.net.ConnectException: Cannot assign requested address
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
at java.net.Socket.connect(Socket.java:529)
at org.apache.zookeeper.server.quorum.Learner.connectToLeader(Learner.java:224)
at org.apache.zookeeper.server.quorum.Observer.observeLeader(Observer.java:69)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:727)
2014-20-03 13:45:33.863 INFO FileSnap [QuorumPeer[myid=3]/0:0:0:0:0:0:0:0:5555] - Reading snapshot /xxx/zk_data/version-2/snapshot.2c00000000
{noformat}

This exception for a while was interspersed with the NPEs but eventually it just was spewing the ConnectionException.

Looking through the code a bit it seems if the FileTxnIterator when initialized cannot find any log files the {{inputStream}} is set to null which causes truncate() to NPE.. I see in 3.4.6 this has been wrapped in a try/finally which closes the iterator.. but i presume that this issue would still remain.

Looking at the system in this state there were 29k+ sockets in CLOSE_WAIT state on the system and looking at a heap dump there were tons of Socket objects waiting for GC (ie not getting properly closed).. this eventually ran the system out of ephemeral ports and hence the ConnectionExceptions..

It would seem that a quick check of {{itr.next()}} prior to attempting truncation would resolve the NPE, but it seems somewhere a connection is not being closed properly when an exception occurs.
381270 No Perforce job exists for this issue. 4 381546
5 years, 38 weeks, 2 days ago
Reviewed
0|i1tou7:
ZooKeeper ZOOKEEPER-1899

zookeeper-cli does not use STDERR and STDOUT correctly to output information

Bug Open Major Unresolved Unassigned Srinath Mantripragada Srinath Mantripragada 20/Mar/14 11:01   20/Mar/14 12:43   3.4.5       0 2   Centos 6, Claudera Hadoop 5 When running a zookeeper-cli, any error or logging information should go to STDERR and the result(s) of the command to STDOUT. For example, let's take the unix 'ls' command:

Unix, STDERR is redirected to '/dev/null' and no results are shown:

{code}
$ ls /aa 2> /dev/null
{code}

zookeeper-cli, everything goes to STDOUT where only the last line should:

{code}
zookeeper-client ls / 2> /dev/null
Connecting to localhost:2181
2014-03-20 14:53:12,220 [myid:] - INFO [main:Environment@100] - Client environment:zookeeper.version=3.4.5-cdh5.0.0-beta-2--1, built on 02/07/2014 18:28 GMT
2014-03-20 14:53:12,227 [myid:] - INFO [main:Environment@100] - Client environment:host.name=node1
2014-03-20 14:53:12,228 [myid:] - INFO [main:Environment@100] - Client environment:java.version=1.7.0_51
2014-03-20 14:53:12,229 [myid:] - INFO [main:Environment@100] - Client environment:java.vendor=Oracle Corporation
2014-03-20 14:53:12,230 [myid:] - INFO [main:Environment@100] - Client environment:java.home=/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.51.x86_64/jre
2014-03-20 14:53:12,231 [myid:] - INFO [main:Environment@100] - Client environment:java.class.path=/usr/lib/zookeeper/bin/../build/classes:/usr/lib/zookeeper/bin/../build/lib/*.jar:/usr/lib/zookeeper/bin/../lib/slf4j-log4j12.jar:/usr/lib/zookeeper/bin/../lib/slf4j-log4j12-1.7.5.jar:/usr/lib/zookeeper/bin/../lib/slf4j-api-1.7.5.jar:/usr/lib/zookeeper/bin/../lib/netty-3.2.2.Final.jar:/usr/lib/zookeeper/bin/../lib/log4j-1.2.15.jar:/usr/lib/zookeeper/bin/../lib/jline-0.9.94.jar:/usr/lib/zookeeper/bin/../zookeeper-3.4.5-cdh5.0.0-beta-2.jar:/usr/lib/zookeeper/bin/../src/java/lib/*.jar:/etc/zookeeper/conf::/etc/zookeeper/conf:/usr/lib/zookeeper/zookeeper.jar:/usr/lib/zookeeper/zookeeper-3.4.5-cdh5.0.0-beta-2.jar:/usr/lib/zookeeper/lib/slf4j-log4j12.jar:/usr/lib/zookeeper/lib/slf4j-api-1.7.5.jar:/usr/lib/zookeeper/lib/log4j-1.2.15.jar:/usr/lib/zookeeper/lib/slf4j-log4j12-1.7.5.jar:/usr/lib/zookeeper/lib/jline-0.9.94.jar:/usr/lib/zookeeper/lib/netty-3.2.2.Final.jar
2014-03-20 14:53:12,232 [myid:] - INFO [main:Environment@100] - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2014-03-20 14:53:12,233 [myid:] - INFO [main:Environment@100] - Client environment:java.io.tmpdir=/tmp
2014-03-20 14:53:12,234 [myid:] - INFO [main:Environment@100] - Client environment:java.compiler=<NA>
2014-03-20 14:53:12,235 [myid:] - INFO [main:Environment@100] - Client environment:os.name=Linux
2014-03-20 14:53:12,235 [myid:] - INFO [main:Environment@100] - Client environment:os.arch=amd64
2014-03-20 14:53:12,236 [myid:] - INFO [main:Environment@100] - Client environment:os.version=2.6.32-431.3.1.el6.x86_64
2014-03-20 14:53:12,237 [myid:] - INFO [main:Environment@100] - Client environment:user.name=hdfs
2014-03-20 14:53:12,238 [myid:] - INFO [main:Environment@100] - Client environment:user.home=/var/lib/hadoop-hdfs
2014-03-20 14:53:12,239 [myid:] - INFO [main:Environment@100] - Client environment:user.dir=/var/lib/hadoop-hdfs
2014-03-20 14:53:12,242 [myid:] - INFO [main:ZooKeeper@438] - Initiating client connection, connectString=localhost:2181 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@5220c1b
2014-03-20 14:53:12,294 [myid:] - INFO [main-SendThread(localhost.localdomain:2181):ClientCnxn$SendThread@966] - Opening socket connection to server localhost.localdomain/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
2014-03-20 14:53:12,305 [myid:] - INFO [main-SendThread(localhost.localdomain:2181):ClientCnxn$SendThread@849] - Socket connection established to localhost.localdomain/127.0.0.1:2181, initiating session
2014-03-20 14:53:12,319 [myid:] - INFO [main-SendThread(localhost.localdomain:2181):ClientCnxn$SendThread@1207] - Session establishment complete on server localhost.localdomain/127.0.0.1:2181, sessionid = 0x144dbe27e1b001d, negotiated timeout = 30000

WATCHER::

WatchedEvent state:SyncConnected type:None path:null
[hadoop-ha, zookeeper]
{code}


For the get command STDOUT and STDERR are inverted:

Results going to STDERR:

{code}
$ zookeeper-client get /hadoop-ha/Redlabnet 1>/dev/null
cZxid = 0x300000027
ctime = Thu Mar 20 14:25:17 UTC 2014
mZxid = 0x300000027
mtime = Thu Mar 20 14:25:17 UTC 2014
pZxid = 0x300000027
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 0
numChildren = 0
{code}


Logs/Errors going to STDOUT:
{code}
$ zookeeper-client get /hadoop-ha/Redlabnet 2>/dev/null
Connecting to localhost:2181
2014-03-20 15:01:22,170 [myid:] - INFO [main:Environment@100] - Client environment:zookeeper.version=3.4.5-cdh5.0.0-beta-2--1, built on 02/07/2014 18:28 GMT
2014-03-20 15:01:22,177 [myid:] - INFO [main:Environment@100] - Client environment:host.name=ip-172-17-0-105.redlabnet.internal
2014-03-20 15:01:22,178 [myid:] - INFO [main:Environment@100] - Client environment:java.version=1.7.0_51
2014-03-20 15:01:22,179 [myid:] - INFO [main:Environment@100] - Client environment:java.vendor=Oracle Corporation
2014-03-20 15:01:22,180 [myid:] - INFO [main:Environment@100] - Client environment:java.home=/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.51.x86_64/jre
2014-03-20 15:01:22,181 [myid:] - INFO [main:Environment@100] - Client environment:java.class.path=/usr/lib/zookeeper/bin/../build/classes:/usr/lib/zookeeper/bin/../build/lib/*.jar:/usr/lib/zookeeper/bin/../lib/slf4j-log4j12.jar:/usr/lib/zookeeper/bin/../lib/slf4j-log4j12-1.7.5.jar:/usr/lib/zookeeper/bin/../lib/slf4j-api-1.7.5.jar:/usr/lib/zookeeper/bin/../lib/netty-3.2.2.Final.jar:/usr/lib/zookeeper/bin/../lib/log4j-1.2.15.jar:/usr/lib/zookeeper/bin/../lib/jline-0.9.94.jar:/usr/lib/zookeeper/bin/../zookeeper-3.4.5-cdh5.0.0-beta-2.jar:/usr/lib/zookeeper/bin/../src/java/lib/*.jar:/etc/zookeeper/conf::/etc/zookeeper/conf:/usr/lib/zookeeper/zookeeper.jar:/usr/lib/zookeeper/zookeeper-3.4.5-cdh5.0.0-beta-2.jar:/usr/lib/zookeeper/lib/slf4j-log4j12.jar:/usr/lib/zookeeper/lib/slf4j-api-1.7.5.jar:/usr/lib/zookeeper/lib/log4j-1.2.15.jar:/usr/lib/zookeeper/lib/slf4j-log4j12-1.7.5.jar:/usr/lib/zookeeper/lib/jline-0.9.94.jar:/usr/lib/zookeeper/lib/netty-3.2.2.Final.jar
2014-03-20 15:01:22,182 [myid:] - INFO [main:Environment@100] - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2014-03-20 15:01:22,182 [myid:] - INFO [main:Environment@100] - Client environment:java.io.tmpdir=/tmp
2014-03-20 15:01:22,183 [myid:] - INFO [main:Environment@100] - Client environment:java.compiler=<NA>
2014-03-20 15:01:22,184 [myid:] - INFO [main:Environment@100] - Client environment:os.name=Linux
2014-03-20 15:01:22,185 [myid:] - INFO [main:Environment@100] - Client environment:os.arch=amd64
2014-03-20 15:01:22,186 [myid:] - INFO [main:Environment@100] - Client environment:os.version=2.6.32-431.3.1.el6.x86_64
2014-03-20 15:01:22,186 [myid:] - INFO [main:Environment@100] - Client environment:user.name=hdfs
2014-03-20 15:01:22,187 [myid:] - INFO [main:Environment@100] - Client environment:user.home=/var/lib/hadoop-hdfs
2014-03-20 15:01:22,188 [myid:] - INFO [main:Environment@100] - Client environment:user.dir=/var/lib/hadoop-hdfs
2014-03-20 15:01:22,191 [myid:] - INFO [main:ZooKeeper@438] - Initiating client connection, connectString=localhost:2181 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@4bb1c978
2014-03-20 15:01:22,242 [myid:] - INFO [main-SendThread(localhost.localdomain:2181):ClientCnxn$SendThread@966] - Opening socket connection to server localhost.localdomain/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
2014-03-20 15:01:22,266 [myid:] - INFO [main-SendThread(localhost.localdomain:2181):ClientCnxn$SendThread@849] - Socket connection established to localhost.localdomain/127.0.0.1:2181, initiating session
2014-03-20 15:01:22,284 [myid:] - INFO [main-SendThread(localhost.localdomain:2181):ClientCnxn$SendThread@1207] - Session establishment complete on server localhost.localdomain/127.0.0.1:2181, sessionid = 0x144dbe27e1b001f, negotiated timeout = 30000

WATCHER::

WatchedEvent state:SyncConnected type:None path:null

{code}


380994 No Perforce job exists for this issue. 0 381272
6 years, 1 week ago 0|i1tn5r:
ZooKeeper ZOOKEEPER-1898

ZooKeeper Java cli shell always returns "0" as exit code

Bug Closed Critical Fixed Abraham Fine Srinath Mantripragada Srinath Mantripragada 20/Mar/14 10:45   17/May/17 23:43 26/Jul/16 19:36 3.4.5 3.5.3, 3.6.0 java client   0 7   CentOS 6, Claudera Hadoop 5 zookeeper-cli always return "0" as exit code whether the command has been successful or not.

Ex:

Unsuccessful:
{code}
-bash-4.1$ zookeeper-client aa
Connecting to localhost:2181
2014-03-20 14:43:01,361 [myid:] - INFO [main:Environment@100] - Client environment:zookeeper.version=3.4.5-cdh5.0.0-beta-2--1, built on 02/07/2014 18:28 GMT
2014-03-20 14:43:01,368 [myid:] - INFO [main:Environment@100] - Client environment:host.name=ip-172-17-0-105.redlabnet.internal
2014-03-20 14:43:01,369 [myid:] - INFO [main:Environment@100] - Client environment:java.version=1.7.0_51
2014-03-20 14:43:01,370 [myid:] - INFO [main:Environment@100] - Client environment:java.vendor=Oracle Corporation
2014-03-20 14:43:01,371 [myid:] - INFO [main:Environment@100] - Client environment:java.home=/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.51.x86_64/jre
2014-03-20 14:43:01,371 [myid:] - INFO [main:Environment@100] - Client environment:java.class.path=/usr/lib/zookeeper/bin/../build/classes:/usr/lib/zookeeper/bin/../build/lib/*.jar:/usr/lib/zookeeper/bin/../lib/slf4j-log4j12.jar:/usr/lib/zookeeper/bin/../lib/slf4j-log4j12-1.7.5.jar:/usr/lib/zookeeper/bin/../lib/slf4j-api-1.7.5.jar:/usr/lib/zookeeper/bin/../lib/netty-3.2.2.Final.jar:/usr/lib/zookeeper/bin/../lib/log4j-1.2.15.jar:/usr/lib/zookeeper/bin/../lib/jline-0.9.94.jar:/usr/lib/zookeeper/bin/../zookeeper-3.4.5-cdh5.0.0-beta-2.jar:/usr/lib/zookeeper/bin/../src/java/lib/*.jar:/etc/zookeeper/conf::/etc/zookeeper/conf:/usr/lib/zookeeper/zookeeper.jar:/usr/lib/zookeeper/zookeeper-3.4.5-cdh5.0.0-beta-2.jar:/usr/lib/zookeeper/lib/slf4j-log4j12.jar:/usr/lib/zookeeper/lib/slf4j-api-1.7.5.jar:/usr/lib/zookeeper/lib/log4j-1.2.15.jar:/usr/lib/zookeeper/lib/slf4j-log4j12-1.7.5.jar:/usr/lib/zookeeper/lib/jline-0.9.94.jar:/usr/lib/zookeeper/lib/netty-3.2.2.Final.jar
2014-03-20 14:43:01,372 [myid:] - INFO [main:Environment@100] - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2014-03-20 14:43:01,373 [myid:] - INFO [main:Environment@100] - Client environment:java.io.tmpdir=/tmp
2014-03-20 14:43:01,374 [myid:] - INFO [main:Environment@100] - Client environment:java.compiler=<NA>
2014-03-20 14:43:01,375 [myid:] - INFO [main:Environment@100] - Client environment:os.name=Linux
2014-03-20 14:43:01,375 [myid:] - INFO [main:Environment@100] - Client environment:os.arch=amd64
2014-03-20 14:43:01,376 [myid:] - INFO [main:Environment@100] - Client environment:os.version=2.6.32-431.3.1.el6.x86_64
2014-03-20 14:43:01,377 [myid:] - INFO [main:Environment@100] - Client environment:user.name=hdfs
2014-03-20 14:43:01,377 [myid:] - INFO [main:Environment@100] - Client environment:user.home=/var/lib/hadoop-hdfs
2014-03-20 14:43:01,378 [myid:] - INFO [main:Environment@100] - Client environment:user.dir=/var/lib/hadoop-hdfs
2014-03-20 14:43:01,382 [myid:] - INFO [main:ZooKeeper@438] - Initiating client connection, connectString=localhost:2181 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@5220c1b
ZooKeeper -server host:port cmd args
connect host:port
get path [watch]
ls path [watch]
set path data [version]
rmr path
delquota [-n|-b] path
quit
printwatches on|off
create [-s] [-e] path data acl
stat path [watch]
close
ls2 path [watch]
history
listquota path
setAcl path acl
getAcl path
sync path
redo cmdno
addauth scheme auth
delete path [version]
setquota -n|-b val path

-bash-4.1$ echo $?
0
{code}

Successful:
{code}
-bash-4.1$ zookeeper-client ls /
Connecting to localhost:2181
2014-03-20 14:43:53,881 [myid:] - INFO [main:Environment@100] - Client environment:zookeeper.version=3.4.5-cdh5.0.0-beta-2--1, built on 02/07/2014 18:28 GMT
2014-03-20 14:43:53,889 [myid:] - INFO [main:Environment@100] - Client environment:host.name=ip-172-17-0-105.redlabnet.internal
2014-03-20 14:43:53,889 [myid:] - INFO [main:Environment@100] - Client environment:java.version=1.7.0_51
2014-03-20 14:43:53,890 [myid:] - INFO [main:Environment@100] - Client environment:java.vendor=Oracle Corporation
2014-03-20 14:43:53,891 [myid:] - INFO [main:Environment@100] - Client environment:java.home=/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.51.x86_64/jre
2014-03-20 14:43:53,892 [myid:] - INFO [main:Environment@100] - Client environment:java.class.path=/usr/lib/zookeeper/bin/../build/classes:/usr/lib/zookeeper/bin/../build/lib/*.jar:/usr/lib/zookeeper/bin/../lib/slf4j-log4j12.jar:/usr/lib/zookeeper/bin/../lib/slf4j-log4j12-1.7.5.jar:/usr/lib/zookeeper/bin/../lib/slf4j-api-1.7.5.jar:/usr/lib/zookeeper/bin/../lib/netty-3.2.2.Final.jar:/usr/lib/zookeeper/bin/../lib/log4j-1.2.15.jar:/usr/lib/zookeeper/bin/../lib/jline-0.9.94.jar:/usr/lib/zookeeper/bin/../zookeeper-3.4.5-cdh5.0.0-beta-2.jar:/usr/lib/zookeeper/bin/../src/java/lib/*.jar:/etc/zookeeper/conf::/etc/zookeeper/conf:/usr/lib/zookeeper/zookeeper.jar:/usr/lib/zookeeper/zookeeper-3.4.5-cdh5.0.0-beta-2.jar:/usr/lib/zookeeper/lib/slf4j-log4j12.jar:/usr/lib/zookeeper/lib/slf4j-api-1.7.5.jar:/usr/lib/zookeeper/lib/log4j-1.2.15.jar:/usr/lib/zookeeper/lib/slf4j-log4j12-1.7.5.jar:/usr/lib/zookeeper/lib/jline-0.9.94.jar:/usr/lib/zookeeper/lib/netty-3.2.2.Final.jar
2014-03-20 14:43:53,893 [myid:] - INFO [main:Environment@100] - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2014-03-20 14:43:53,894 [myid:] - INFO [main:Environment@100] - Client environment:java.io.tmpdir=/tmp
2014-03-20 14:43:53,894 [myid:] - INFO [main:Environment@100] - Client environment:java.compiler=<NA>
2014-03-20 14:43:53,895 [myid:] - INFO [main:Environment@100] - Client environment:os.name=Linux
2014-03-20 14:43:53,896 [myid:] - INFO [main:Environment@100] - Client environment:os.arch=amd64
2014-03-20 14:43:53,897 [myid:] - INFO [main:Environment@100] - Client environment:os.version=2.6.32-431.3.1.el6.x86_64
2014-03-20 14:43:53,897 [myid:] - INFO [main:Environment@100] - Client environment:user.name=hdfs
2014-03-20 14:43:53,898 [myid:] - INFO [main:Environment@100] - Client environment:user.home=/var/lib/hadoop-hdfs
2014-03-20 14:43:53,899 [myid:] - INFO [main:Environment@100] - Client environment:user.dir=/var/lib/hadoop-hdfs
2014-03-20 14:43:53,902 [myid:] - INFO [main:ZooKeeper@438] - Initiating client connection, connectString=localhost:2181 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@5a9e40d2
2014-03-20 14:43:53,953 [myid:] - INFO [main-SendThread(localhost.localdomain:2181):ClientCnxn$SendThread@966] - Opening socket connection to server localhost.localdomain/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
2014-03-20 14:43:53,963 [myid:] - INFO [main-SendThread(localhost.localdomain:2181):ClientCnxn$SendThread@849] - Socket connection established to localhost.localdomain/127.0.0.1:2181, initiating session
2014-03-20 14:43:53,977 [myid:] - INFO [main-SendThread(localhost.localdomain:2181):ClientCnxn$SendThread@1207] - Session establishment complete on server localhost.localdomain/127.0.0.1:2181, sessionid = 0x144dbe27e1b0013, negotiated timeout = 30000

WATCHER::

WatchedEvent state:SyncConnected type:None path:null
[hadoop-ha, zookeeper]

-bash-4.1$ echo $?
0

{code}
380988 No Perforce job exists for this issue. 3 381266
3 years, 34 weeks, 2 days ago
Reviewed
0|i1tn4f:
ZooKeeper ZOOKEEPER-1897

ZK Shell/Cli not processing commands

Bug Resolved Major Fixed Michael Stack Cameron Gandevia Cameron Gandevia 18/Mar/14 19:10   20/May/17 19:07 03/Apr/14 18:41 3.4.6 3.4.7, 3.5.0 java client, scripts   0 7   When running zookeeper 3.4.5 I was able to run commands using zkCli such as

zkCli.sh -server 127.0.0.1:2182 ls /
zkCli.sh -server 127.0.0.1:2182 get /blah

After upgrading to 3.4.6 these commands no longer work.

I think issue https://issues.apache.org/jira/browse/ZOOKEEPER-1535 was the reason the commands were running in previous versions.

It looks like the client exits when a command is present.

{code:title=ZooKeeperMain.java}
void run() throws KeeperException, IOException, InterruptedException {
if (cl.getCommand() == null) {
System.out.println("Welcome to ZooKeeper!");

boolean jlinemissing = false;
// only use jline if it's in the classpath
try {
Class consoleC = Class.forName("jline.ConsoleReader");
Class completorC =
Class.forName("org.apache.zookeeper.JLineZNodeCompletor");

System.out.println("JLine support is enabled");

Object console =
consoleC.getConstructor().newInstance();

Object completor =
completorC.getConstructor(ZooKeeper.class).newInstance(zk);
Method addCompletor = consoleC.getMethod("addCompletor",
Class.forName("jline.Completor"));
addCompletor.invoke(console, completor);

String line;
Method readLine = consoleC.getMethod("readLine", String.class);
while ((line = (String)readLine.invoke(console, getPrompt())) != null) {
executeLine(line);
}
} catch (ClassNotFoundException e) {
LOG.debug("Unable to start jline", e);
jlinemissing = true;
} catch (NoSuchMethodException e) {
LOG.debug("Unable to start jline", e);
jlinemissing = true;
} catch (InvocationTargetException e) {
LOG.debug("Unable to start jline", e);
jlinemissing = true;
} catch (IllegalAccessException e) {
LOG.debug("Unable to start jline", e);
jlinemissing = true;
} catch (InstantiationException e) {
LOG.debug("Unable to start jline", e);
jlinemissing = true;
}

if (jlinemissing) {
System.out.println("JLine support is disabled");
BufferedReader br =
new BufferedReader(new InputStreamReader(System.in));

String line;
while ((line = br.readLine()) != null) {
executeLine(line);
}
}
}
}
{code}
380622 No Perforce job exists for this issue. 3 380901
5 years, 50 weeks, 6 days ago 0|i1tkvj:
ZooKeeper ZOOKEEPER-1896

Reconfig error messages when upgrading from 3.4.6 to 3.5.0

Bug Open Major Unresolved Unassigned Raúl Gutiérrez Segalés Raúl Gutiérrez Segalés 17/Mar/14 12:41   05/Feb/20 07:16   3.5.0 3.7.0, 3.5.8 server   0 5   When upgrading from 3.4.6 (rc0 actually) to 3.5.0 (trunk as of two weeks ago actually) I got this error message:

{noformat}
2014-02-26 22:12:15,446 - ERROR [WorkerReceiver[myid=4]] - Something went wrong while processing config received from 3
{noformat}

According to [~fpj]:

bq. I think you’re right that the reconfig error is harmless, but we shouldn’t be getting it. The problem is that it is not detecting that we are in backward compatibility mode. We need to fix it for 3.5.0 and perhaps ZOOKEEPER-1805 is the right place for doing it.

cc: [~shralex]
380268 No Perforce job exists for this issue. 0 380551
1 year, 17 weeks, 1 day ago 0|i1tiqn:
ZooKeeper ZOOKEEPER-1895

update all notice files, copyright, etc... with the new year - 2014

Bug Resolved Blocker Fixed Michi Mutsuzaki Patrick D. Hunt Patrick D. Hunt 16/Mar/14 12:35   20/May/14 07:09 16/May/14 13:58 3.4.7, 3.5.0 3.4.7, 3.5.0     0 3   From a note on the list:

Hi folks!

This is a reminder to update the year in the NOTICE files from 2013 (or older) to 2014.

From a legal POV this is not that important as some say.
But nonetheless it's good to update the year.

LieGrue,
strub
380098 No Perforce job exists for this issue. 1 380382
5 years, 44 weeks, 2 days ago 0|i1thp3:
ZooKeeper ZOOKEEPER-1894

ObserverTest.testObserver fails consistently

Bug Resolved Major Fixed Michi Mutsuzaki Michi Mutsuzaki Michi Mutsuzaki 13/Mar/14 22:43   27/Mar/14 21:20 27/Mar/14 20:59 3.5.0 3.5.0 quorum   0 7   ubuntu 13.10
Server environment:java.version=1.7.0_51
Server environment:java.vendor=Oracle Corporation
ObserverTest.testObserver fails consistently on my box. It looks like the observer (myid:3) calls QuorumPeer.getQuorumVerifier() in a tight loop, and the leader (myid:2) is not getting enough CPU time to synchronize with the follower and the observer. The test passes if I increase ClientBase.CONNECTION_TIMEOUT from 30 seconds to 120 seconds. I'll attach a log file. 379750 No Perforce job exists for this issue. 4 380035
5 years, 51 weeks, 6 days ago 0|i1tfkf:
ZooKeeper ZOOKEEPER-1893

automake: use serial-tests option

Bug Resolved Minor Fixed Michi Mutsuzaki Michi Mutsuzaki Michi Mutsuzaki 12/Mar/14 23:46   17/Apr/15 14:11 14/Mar/15 21:10   3.5.1, 3.6.0 c client   0 8   automake switched to run tests in parallel by default in 1.13, but zktest-st and zktest-mt can't run in parallel. We can use the serial-tests option to run tests serially but this option was introduced in automake 1.12. I don't know which version of automake buidbot has. I'll upload the patch and see. 379499 No Perforce job exists for this issue. 1 379790
4 years, 48 weeks, 6 days ago 0|i1te1z:
ZooKeeper ZOOKEEPER-1892

addrvec_next gets called twice when failing over to the next server

Bug Resolved Major Duplicate Unassigned Michi Mutsuzaki Michi Mutsuzaki 12/Mar/14 22:06   12/Mar/14 23:41 12/Mar/14 23:41   3.5.0 c client   0 3   zookeeper_interest() already calls zoo_cycle_next_server() when the socket is set to -1, so we shouldn't call addrvec_next in handle_error. This causes the next server to get skipped. Zookeeper_simpleSystem::testFirstServerDown fails unless the client gets connected to the server during the first round because the client keeps skipping the second server after the first round. 379491 No Perforce job exists for this issue. 1 379782
6 years, 2 weeks ago 0|i1te07:
ZooKeeper ZOOKEEPER-1891

StaticHostProviderTest.testUpdateLoadBalancing times out

Bug Resolved Major Fixed Michi Mutsuzaki Michi Mutsuzaki Michi Mutsuzaki 10/Mar/14 19:33   20/May/14 07:09 10/May/14 06:46 3.5.0 3.5.0 java client   0 5   ubuntu 13.10
Server environment:java.version=1.7.0_51
Server environment:java.vendor=Oracle Corporation
StaticHostProviderTest.testUpdateLoadBalancing is consistently timing out on my box. I'll attach a log file. 378246 No Perforce job exists for this issue. 3 378538
5 years, 44 weeks, 2 days ago 0|i1t6cf:
ZooKeeper ZOOKEEPER-1890

unclosed FileOutputStream in FileTxnLog.rollLog

Bug Open Major Unresolved Unassigned Andrew Gaul Andrew Gaul 27/Feb/14 20:21   14/Feb/15 00:13           0 4   When calling rollLog, FileTxnLog flushes but does not close its FileOutputStream, leaking a file descriptor. 376312 No Perforce job exists for this issue. 1 376608
5 years, 5 weeks, 5 days ago 0|i1sugn:
ZooKeeper ZOOKEEPER-1889

Implement Top-N sort operator

Bug Resolved Major Not A Problem Unassigned Steven Phillips Steven Phillips 25/Feb/14 19:38   25/Feb/14 19:52 25/Feb/14 19:52         0 2   When, for example, doing an order by with a limit, if limit << total, it would be much more efficient to maintain a priority queue instead of sorting the entire data set.

In most cases, this will greatly reduce the number of comparisons, since most incoming records will not fall in the Top N, and thus will only require a single comparison operation. Incoming records that are in the Top-N will require at most log N comparisons.

This will also allow periodic purging of record batches, reducing memory requirements.
375742 No Perforce job exists for this issue. 0 376038
6 years, 4 weeks, 1 day ago 0|i1sqyf:
ZooKeeper ZOOKEEPER-1888

ZkCli.cmd commands fail with "'java' is not recognized as an internal or external command"

Bug Resolved Major Fixed Ivan Mitic Ivan Mitic Ivan Mitic 25/Feb/14 14:07   13/Mar/14 22:55 13/Mar/14 22:55 3.4.5 3.4.7, 3.5.0     0 5   Windows This appears to be a bug in ZkCli.cmd as it does not try to locate java using the JAVA_HOME environment variable.

Will post the patch soon.
375661 No Perforce job exists for this issue. 3 375957
6 years, 1 week, 6 days ago 0|i1sqgf:
ZooKeeper ZOOKEEPER-1887

C implementation of removeWatches

New Feature Resolved Major Fixed Raúl Gutiérrez Segalés Raúl Gutiérrez Segalés Raúl Gutiérrez Segalés 23/Feb/14 20:00   16/Dec/18 09:50 16/Apr/14 02:16   3.5.0 c client   1 8   This is equivalent for ZOOKEEPER-442's Java impl. remove_watches 375303 No Perforce job exists for this issue. 3 375599
5 years, 49 weeks, 1 day ago 0|i1so93:
ZooKeeper ZOOKEEPER-1886

Exception in Follower.followLeader() where Leader is still running, can make that follower hang in LeaderElection

Bug Open Major Unresolved Unassigned Vinayakumar B Vinayakumar B 21/Feb/14 00:35   21/Feb/14 00:35       server   0 2   SocketTimeoutException in {{Follower#followLeader()}} where the leader is successfully running can make this follower not able to rejoin the quorum.

Analysis:
1. SocketTimeoutException in below code, will make follower to stop following (Not process shutdown) and try to participate in leader election again.
{code} while (self.isRunning()) {
readPacket(qp);
processPacket(qp);
}{code}

2. At the time of leader election {{FastLeaderElection#logicalclock}} will be incremented at only follower side, and this is more than electionEpoch of the leader.

3. Notification from the Leader will get ignored and from this follower notifications will be continously sent and again ignored.

374944 No Perforce job exists for this issue. 0 375243
6 years, 4 weeks, 6 days ago 0|i1sm1z:
ZooKeeper ZOOKEEPER-1885

Znodes deletable by anyone without having the rights to do so

Bug Resolved Major Not A Problem Unassigned Behar Veliqi Behar Veliqi 20/Feb/14 10:55   21/Feb/14 04:52 21/Feb/14 04:52 3.4.5       0 2   Ubuntu 12.04 LTS 64-bit Hi,

I'm not really sure if this is a bug or a misunderstanding on my part, but I have the problem that, when I create a znode with an ACL as follows:

{noformat}
[zk: localhost:2181(CONNECTED) 60] create /anode "somecontent" digest:'user:IAEttLCxci/qWhKN2QJ6u1nrQgw=':cdrwa
Created /anode
[zk: localhost:2181(CONNECTED) 61] getAcl /anode
'digest,''user:IAEttLCxci/qWhKN2QJ6u1nrQgw='
: cdrwa
{noformat}

I am not able to read or update the content of the node, as it should be:

{noformat}
[zk: localhost:2181(CONNECTED) 62] get /anode
Authentication is not valid : /anode
[zk: localhost:2181(CONNECTED) 63] set /anode "update"
Authentication is not valid : /anode
{noformat}

But everyone without being authenticated can delete the node:

{noformat}
[zk: localhost:2181(CONNECTED) 64] delete /anode
[zk: localhost:2181(CONNECTED) 65] get /anode
Node does not exist: /anode
{noformat}

Is this a bug or is there a way to set the ACL so that only the user having the credentials can delete the znode?

374756 No Perforce job exists for this issue. 0 375056
6 years, 4 weeks, 6 days ago 0|i1skwf:
ZooKeeper ZOOKEEPER-1884

zkCli silently ignores commands with missing parameters

Bug Open Minor Unresolved Raúl Gutiérrez Segalés Flavio Paiva Junqueira Flavio Paiva Junqueira 20/Feb/14 06:58   22/Jun/18 00:49   3.4.6, 3.4.11       0 4   Apparently, we have fixed this in trunk, but not in the 3.4 branch. When we pass only the path to create, the command is not executed because it expects an additional parameter and there is no error message because the create command exists. 374697 No Perforce job exists for this issue. 1 374997
3 years, 30 weeks, 1 day ago 0|i1skjb:
ZooKeeper ZOOKEEPER-1883

C client unit test failures

Bug Resolved Minor Fixed Raúl Gutiérrez Segalés Abhiraj Butala Abhiraj Butala 17/Feb/14 22:30   13/Mar/14 18:08 13/Mar/14 16:58   3.5.0 c client, tests   0 5   I am seeing unit test failure for c client after I do 'make check' as shown below. The failure is pretty consistent, but does not happen always. This is on latest check-out of zookeeper trunk.

------------------
Zookeeper_simpleSystem::testAsyncWatcherAutoReset ZooKeeper server started : elapsed 9640 : OK
Zookeeper_simpleSystem::testDeserializeString : elapsed 0 : OK
Zookeeper_simpleSystem::testFirstServerDown : elapsed 1007 : OK
Zookeeper_simpleSystem::testNullData : elapsed 1028 : OK
Zookeeper_simpleSystem::testIPV6 : elapsed 1008 : OK
Zookeeper_simpleSystem::testCreate : elapsed 1016 : OK
Zookeeper_simpleSystem::testPath : elapsed 1083 : OK
Zookeeper_simpleSystem::testPathValidation : elapsed 1046 : OK
Zookeeper_simpleSystem::testPing : elapsed 17301 : OK
Zookeeper_simpleSystem::testAcl : elapsed 1018 : OK
Zookeeper_simpleSystem::testChroot : elapsed 3057 : OK
Zookeeper_simpleSystem::testAuth ZooKeeper server started ZooKeeper server started : elapsed 29357 : OK
Zookeeper_simpleSystem::testHangingClient : elapsed 1037 : OK
Zookeeper_simpleSystem::testWatcherAutoResetWithGlobal ZooKeeper server started ZooKeeper server started ZooKeeper server started : elapsed 12983 : OK
Zookeeper_simpleSystem::testWatcherAutoResetWithLocal ZooKeeper server started ZooKeeper server started ZooKeeper server started : elapsed 13028 : OK
Zookeeper_simpleSystem::testGetChildren2 : elapsed 1031 : OK
Zookeeper_simpleSystem::testLastZxid : assertion : elapsed 2514
Zookeeper_watchers::testDefaultSessionWatcher1 : elapsed 52 : OK
Zookeeper_watchers::testDefaultSessionWatcher2 : elapsed 3 : OK
Zookeeper_watchers::testObjectSessionWatcher1 : elapsed 52 : OK
Zookeeper_watchers::testObjectSessionWatcher2 : elapsed 54 : OK
Zookeeper_watchers::testNodeWatcher1 : elapsed 55 : OK
Zookeeper_watchers::testChildWatcher1 : elapsed 3 : OK
Zookeeper_watchers::testChildWatcher2 : elapsed 3 : OK
tests/TestClient.cc:1281: Assertion: equality assertion failed [Expected: 1239, Actual : 1238]
Failures !!!
Run: 70 Failure total: 1 Failures: 1 Errors: 0
FAIL: zktest-mt
==========================================
1 of 2 tests failed
Please report to user@zookeeper.apache.org
==========================================
make[1]: *** [check-TESTS] Error 1
make[1]: Leaving directory `/home/abutala/work/zk/zookeeper-trunk/src/c'
make: *** [check-am] Error 2
------------------

$ uname -a
Linux abutala-vBox 3.8.0-35-generic #50~precise1-Ubuntu SMP Wed Dec 4 17:25:51 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux

374131 No Perforce job exists for this issue. 1 374431
6 years, 2 weeks ago 0|i1sh1j:
ZooKeeper ZOOKEEPER-1881

Shutdown server immediately upon PrivilegedActionException

Improvement Patch Available Major Unresolved Ding Yuan Ding Yuan Ding Yuan 13/Feb/14 18:53   05/Feb/20 07:11   3.4.5 3.7.0, 3.5.8 server   0 3   It seems when an SaslServer cannot be created due to a PriviledgedActionException, it is better to shutdown the server immediately instead of letting it to propagate. The current behaviour will just set ServerCncx.zooKeeperSaslServer to null, and later every time when an SASL request comes in it will be rejected. If we already detect the loophole early, we should just reject it early.

{noformat}
private SaslServer createSaslServer(final Login login) {
catch (PrivilegedActionException e) {
// TODO: exit server at this point(?)
LOG.error("Zookeeper Quorum member experienced a PrivilegedActionException exception while creating a SaslServer using a JAAS principal context:" + e);
e.printStackTrace();
}
{noformat}

For what it is worth, attaching an attempt to patch it. The idea of the patch is to propagate this PrivilegedActionException to ServerCnxnFactory and shut down all the connections and server. Not sure if this is the right way to solve it. Any comments are appreciated!

Also in the patch are two additional logging on two unlogged exceptions.
373619 No Perforce job exists for this issue. 1 373919
3 years, 39 weeks, 2 days ago 0|i1sdw7:
ZooKeeper ZOOKEEPER-1879

improve the correctness checking of txn log replay

Improvement Open Major Unresolved Unassigned Patrick D. Hunt Patrick D. Hunt 10/Feb/14 15:46   05/Feb/20 07:16   3.4.6, 3.5.0 3.7.0, 3.5.8 server   0 2   In ZOOKEEPER-1573 we decided to fix an issue by relaxing some of the checking. Specifically when the sequence of txns is as follows:

* zxid 1: create /prefix/a
* zxid 2: create /prefix/a/b
* zxid 3: delete /prefix/a/b
* zxid 4: delete /prefix/a

the log may fail to replay.

We addressed this by relaxing a check, which is essentially invalid for this case, but is important in finding corruptions of the datastore.

We should add this check back with proper validation of correctness.
372858 No Perforce job exists for this issue. 0 373162
6 years, 6 weeks, 2 days ago 0|i1s98n:
ZooKeeper ZOOKEEPER-1878

Inconsistent behavior in autocreation of dataDir and dataLogDir

Bug Resolved Major Fixed Rakesh Radhakrishnan Rakesh Radhakrishnan Rakesh Radhakrishnan 08/Feb/14 23:18   27/Mar/14 07:10 26/Mar/14 22:02 3.4.5 3.4.7, 3.5.0 quorum   0 5   During the startup if dataDir is not exists server will auto create this. But when user specifies different dataLogDir location which doesn't exists the server will validate and startup will fail.

{code}
org.apache.zookeeper.server.quorum.QuorumPeerConfig$ConfigException: Error processing build\test3085582797504170966.junit.dir\zoo.cfg
at org.apache.zookeeper.server.quorum.QuorumPeerConfig.parse(QuorumPeerConfig.java:123)
at org.apache.zookeeper.server.ServerConfig.parse(ServerConfig.java:79)
at org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:81)
at org.apache.zookeeper.server.ZooKeeperServerMainTest$MainThread.run(ZooKeeperServerMainTest.java:92)
Caused by: java.lang.IllegalArgumentException: dataLogDir build/test3085582797504170966.junit.dir/data_txnlog is missing.
at org.apache.zookeeper.server.quorum.QuorumPeerConfig.parseProperties(QuorumPeerConfig.java:253)
at org.apache.zookeeper.server.quorum.QuorumPeerConfig.parse(QuorumPeerConfig.java:119)
... 3 more
{code}
372647 No Perforce job exists for this issue. 5 372951
6 years ago 0|i1s7y7:
ZooKeeper ZOOKEEPER-1877

Malformed ACL Id can crash server with skipACL=yes

Bug Resolved Critical Fixed Chris Chen Chris Chen Chris Chen 06/Feb/14 18:13   25/Jul/14 07:25 24/Jul/14 19:32 3.5.0 3.5.0 server   0 4   Because of the way fixupACL is written in PrepRequestProcessor, a request that feeds in an ACL with null members in the Id will cause a server with skipACL=yes to crash.

A patch will be provided that re-introduces checks for well-formed ACLs even if skipACL is enabled.
372277 No Perforce job exists for this issue. 1 372581
5 years, 34 weeks, 6 days ago
Reviewed
0|i1s5of:
ZooKeeper ZOOKEEPER-1876

Add support for installing windows services in .cmd scripts

Improvement Open Major Unresolved Enis Soztutar Enis Soztutar Enis Soztutar 06/Feb/14 16:14   05/Feb/20 07:17     3.7.0, 3.5.8 scripts   0 2   On windows, daemons can be installed as windows services during installation, so that they can be managed using the standard "service" commands and UI.

For this, we have to generate an XML file describing the command line program, and arguments. We can add support for --service parameter passed to bin/zkServer.cmd so that it will output the XML for the service instead of running the command.

Hadoop and HBase has the same syntax and mechanics (see https://github.com/apache/hbase/blob/trunk/bin/hbase.cmd#L73)
372246 No Perforce job exists for this issue. 1 372550
6 years, 7 weeks ago 0|i1s5hj:
ZooKeeper ZOOKEEPER-1875

NullPointerException in ClientCnxn$EventThread.processEvent

Bug Open Minor Unresolved Jerry He Jerry He Jerry He 02/Feb/14 23:06   05/Feb/20 07:16   3.4.5, 3.4.10 3.7.0, 3.5.8 java client   3 11   We've been seeing NullPointerException while working on HBase:

{code}
14/01/30 22:15:25 INFO zookeeper.ZooKeeper: Client environment:user.dir=/home/biadmin/hbase-trunk
14/01/30 22:15:25 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=hdtest009:2181 sessionTimeout=90000 watcher=null
14/01/30 22:15:25 INFO zookeeper.ClientCnxn: Opening socket connection to server hdtest009/9.30.194.18:2181. Will not attempt to authenticate using SASL (Unable to locate a login configuration)
14/01/30 22:15:25 INFO zookeeper.ClientCnxn: Socket connection established to hdtest009/9.30.194.18:2181, initiating session
14/01/30 22:15:25 INFO zookeeper.ClientCnxn: Session establishment complete on server hdtest009/9.30.194.18:2181, sessionid = 0x143986213e67e48, negotiated timeout = 60000
14/01/30 22:15:25 ERROR zookeeper.ClientCnxn: Error while calling watcher
java.lang.NullPointerException
at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
{code}

The reason is the watcher is null in this part of the code:
{code}
private void processEvent(Object event) {
try {
if (event instanceof WatcherSetEventPair) {
// each watcher will process the event
WatcherSetEventPair pair = (WatcherSetEventPair) event;
for (Watcher watcher : pair.watchers) {
try {
watcher.process(pair.event);
} catch (Throwable t) {
LOG.error("Error while calling watcher ", t);
}
}
{code}
371363 No Perforce job exists for this issue. 3 371666
11 weeks ago 0|i1s02v:
ZooKeeper ZOOKEEPER-1874

ZOOKEEPER-1833 Add proper teardown/cleanups in ReconfigTest to shutdown quorumpeer

Sub-task Resolved Major Fixed Unassigned Rakesh Radhakrishnan Rakesh Radhakrishnan 01/Feb/14 23:58   11/Feb/14 21:29 11/Feb/14 21:29   3.5.0 tests   0 4   This jira to provide proper cleanups in ReconfigTest test cases. 371290 No Perforce job exists for this issue. 2 371593
6 years, 6 weeks, 1 day ago 0|i1rzmn:
ZooKeeper ZOOKEEPER-1873

ZOOKEEPER-1833 Unnecessarily InstanceNotFoundException is coming when unregister failed jmxbeans

Sub-task Closed Major Fixed Rakesh Radhakrishnan Rakesh Radhakrishnan Rakesh Radhakrishnan 29/Jan/14 16:06   13/Mar/14 14:16 12/Feb/14 06:17   3.4.6, 3.5.0 server   0 4   MBeanRegistry#register is keeping the beans which are failed to complete the registration. During unregistration time, these failed beans will results in following exception.
{code}
[junit] 2014-01-29 08:34:56,667 [myid:] - WARN [main:MBeanRegistry@134] - Error during unregister
[junit] javax.management.InstanceNotFoundException: org.apache.ZooKeeperService:name0=StandaloneServer_port-1
[junit] at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean(DefaultMBeanServerInterceptor.java:1095)
[junit] at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.exclusiveUnregisterMBean(DefaultMBeanServerInterceptor.java:427)
[junit] at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.unregisterMBean(DefaultMBeanServerInterceptor.java:415)
[junit] at com.sun.jmx.mbeanserver.JmxMBeanServer.unregisterMBean(JmxMBeanServer.java:536)
[junit] at org.apache.zookeeper.jmx.MBeanRegistry.unregister(MBeanRegistry.java:115)
{code}
370731 No Perforce job exists for this issue. 1 371042
6 years, 2 weeks ago 0|i1rw8n:
ZooKeeper ZOOKEEPER-1872

ZOOKEEPER-1833 QuorumPeer is not shutdown in few cases

Sub-task Closed Major Fixed Rakesh Radhakrishnan Rakesh Radhakrishnan Rakesh Radhakrishnan 29/Jan/14 15:19   21/Jul/16 16:18 03/Nov/15 02:26   3.4.7, 3.5.2, 3.6.0     0 7   Few cases are leaving quorumpeer running after the test case execution. Needs proper teardown for these. test 370717 No Perforce job exists for this issue. 15 371028
4 years, 20 weeks, 2 days ago 0|i1rw5r:
ZooKeeper ZOOKEEPER-1871

Add an option to zkCli to wait for connection before executing commands

Improvement Patch Available Major Unresolved Takashi Ohnishi Vinayakumar B Vinayakumar B 29/Jan/14 01:47   05/Feb/20 07:11   3.4.5 3.7.0, 3.5.8     0 3   Add an option to zkCli to wait for connection before executing any commands.
This is helpful for the execution of inline commands.

We are having some scripts to create/delete znodes through commandline. But if getting the connection delays due to one of the node down, then command will fail with connectionloss even though quorum is available.

So I propose a commandline option (similar to -server and -timeout) "-waitforconnection" to wait for the connection before executing any commands.
370559 No Perforce job exists for this issue. 4 370869
3 years, 39 weeks, 2 days ago Change zkCli.sh so as to wait until connection to quorum is done.
Add an option -waitforconnection for cmd args option to set this timeout to be same with session timeout.
0|i1rv7b:
ZooKeeper ZOOKEEPER-1870

flakey test in StandaloneDisabledTest.startSingleServerTest

Bug Resolved Blocker Fixed Helen Hastings Patrick D. Hunt Patrick D. Hunt 28/Jan/14 13:45   28/Aug/14 12:50 28/Aug/14 12:49 3.4.6 3.5.0 tests   0 9   I'm seeing lots of the following failure. Seems like a flakey test (passes every so often).

{noformat}
junit.framework.AssertionFailedError: client could not connect to reestablished quorum: giving up after 30+ seconds.
at org.apache.zookeeper.test.ReconfigTest.testNormalOperation(ReconfigTest.java:143)
at org.apache.zookeeper.server.quorum.StandaloneDisabledTest.startSingleServerTest(StandaloneDisabledTest.java:75)
at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:52)
{noformat}

I've found 3 problems:

1. QuorumCnxManager.Listener.run() leaks the socket depending on when the shutdown flag gets set.
2. QuorumCnxManager.halt() doesn't wait for the listener to terminate.
3. QuorumPeer.shuttingDownLE flag doesn't get reset when restarting the leader election.
370440 No Perforce job exists for this issue. 4 370761
5 years, 39 weeks ago 0|i1rujj:
ZooKeeper ZOOKEEPER-1869

zk server falling apart from quorum due to connection loss and couldn't connect back

Bug Resolved Critical Abandoned Unassigned Deepak Jagtap Deepak Jagtap 27/Jan/14 14:43   03/Mar/16 00:55 03/Mar/16 00:55 3.5.0   quorum   0 16   Using CentOS6 for running these zookeeper servers We have deployed zookeeper version 3.5.0.1515976, with 3 zk servers in the quorum.
The problem we are facing is that one zookeeper server in the quorum falls apart, and never becomes part of the cluster until we restart zookeeper server on that node.

Our interpretation from zookeeper logs on all nodes is as follows:
(For simplicity assume S1=> zk server1, S2 => zk server2, S3 => zk server 3)
Initially S3 is the leader while S1 and S2 are followers.

S2 hits 46 sec latency while fsyncing write ahead log and results in loss of connection with S3.
S3 in turn prints following error message:

Unexpected exception causing shutdown while sock still open
java.net.SocketTimeoutException: Read timed out
Stack trace
******* GOODBYE /169.254.1.2:47647(S2) ********

S2 in this case closes connection with S3(leader) and shuts down follower with following log messages:
Closing connection to leader, exception during packet send
java.net.SocketException: Socket close
Follower@194] - shutdown called
java.lang.Exception: shutdown Follower

After this point S3 could never reestablish connection with S2 and leader election mechanism keeps failing. S3 now keeps printing following message repeatedly:
Cannot open channel to 2 at election address /169.254.1.2:3888
java.net.ConnectException: Connection refused.

While S3 is in this state, S2 repeatedly keeps printing following message:
INFO [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:60667
Exception causing close of session 0x0: ZooKeeperServer not running
Closed socket connection for client /127.0.0.1:60667 (no session established for client)

Leader election never completes successfully and causing S2 to fall apart from the quorum.
S2 was out of quorum for almost 1 week.

While debugging this issue, we found out that both election and peer connection ports on S2 can't be telneted from any of the node (S1, S2, S3). Network connectivity is not the issue. Later, we restarted the ZK server S2 (service zookeeper-server restart) -- now we could telnet to both the ports and S2 joined the ensemble after a leader election attempt.
Any idea what might be forcing S2 to get into a situation where it won't accept any connections on the leader election and peer connection ports?
370195 No Perforce job exists for this issue. 0 370497
4 years, 3 weeks ago 0|i1rsyf:
ZooKeeper ZOOKEEPER-1868

ZOOKEEPER-1833 Server not coming back up in QuorumZxidSyncTest

Sub-task Resolved Major Cannot Reproduce Unassigned Flavio Paiva Junqueira Flavio Paiva Junqueira 27/Jan/14 10:08   24/Aug/15 03:57 24/Aug/15 03:57   3.4.7     0 5   We got this stack trace:

{noformat}
[junit] 2014-01-27 09:14:08,481 [myid:] - INFO [main:ZKTestCase$1@65] - FAILED testLateLogs
[junit] java.lang.AssertionError: waiting for server up
[junit] at org.junit.Assert.fail(Assert.java:91)
[junit] at org.junit.Assert.assertTrue(Assert.java:43)
[junit] at org.apache.zookeeper.test.QuorumBase.startServers(QuorumBase.java:188)
[junit] at org.apache.zookeeper.test.QuorumBase.startServers(QuorumBase.java:113)
[junit] at org.apache.zookeeper.test.QuorumZxidSyncTest.testLateLogs(QuorumZxidSyncTest.java:116)
{noformat}

which occurs here, when we stop the servers and restart them.

{noformat}
qb.shutdownServers();
qb.startServers();
{noformat}
370136 No Perforce job exists for this issue. 1 370438
4 years, 30 weeks, 3 days ago 0|i1rslb:
ZooKeeper ZOOKEEPER-1867

ZOOKEEPER-1833 Bug in ZkDatabaseCorruptionTest

Sub-task Closed Major Fixed Flavio Paiva Junqueira Flavio Paiva Junqueira Flavio Paiva Junqueira 26/Jan/14 21:15   13/Mar/14 14:17 27/Jan/14 08:39   3.4.6, 3.5.0 tests   0 3   If I'm reading the test case testCorruption right, it seems to depend on server 5 being elected, but if it is not the case, then it fails waiting for a server to be up. 370050 No Perforce job exists for this issue. 2 370352
6 years, 2 weeks ago 0|i1rs27:
ZooKeeper ZOOKEEPER-1866

ZOOKEEPER-1833 ClientBase#createClient is failing frequently

Sub-task Resolved Major Not A Problem Germán Blanco Rakesh Radhakrishnan Rakesh Radhakrishnan 22/Jan/14 06:09   19/May/15 04:23 19/May/15 04:23 3.4.5 3.4.7 tests   1 4   Following failure pattern has been observed many times in windows build. After creating the zookeeper client, the respective connection bean is not available in the jmx beans and is failing the tests.

{code}
[junit] 2014-01-22 08:58:22,625 [myid:] - INFO [main:ZKTestCase$1@65] - FAILED testInvalidVersion
[junit] junit.framework.AssertionFailedError: expected [0x143b92b03330000] expected:<1> but was:<0>
[junit] at junit.framework.Assert.fail(Assert.java:47)
[junit] at junit.framework.Assert.failNotEquals(Assert.java:283)
[junit] at junit.framework.Assert.assertEquals(Assert.java:64)
[junit] at junit.framework.Assert.assertEquals(Assert.java:195)
[junit] at org.apache.zookeeper.test.JMXEnv.ensureAll(JMXEnv.java:124)
[junit] at org.apache.zookeeper.test.ClientBase.createClient(ClientBase.java:191)
[junit] at org.apache.zookeeper.test.ClientBase.createClient(ClientBase.java:171)
[junit] at org.apache.zookeeper.test.ClientBase.createClient(ClientBase.java:156)
[junit] at org.apache.zookeeper.test.ClientBase.createClient(ClientBase.java:149)
[junit] at org.apache.zookeeper.test.MultiTransactionTest.setUp(MultiTransactionTest.java:60)
{code}
369198 No Perforce job exists for this issue. 1 369503
4 years, 44 weeks, 2 days ago 0|i1rmuf:
ZooKeeper ZOOKEEPER-1865

Fix retry logic in Learner.connectToLeader()

Bug Reopened Major Unresolved Edward Carter Thawan Kooburat Thawan Kooburat 21/Jan/14 19:51   05/Feb/20 07:16     3.7.0, 3.5.8 server   0 10   We discovered a long leader election time today in one of our prod ensemble.

Here is the description of the event.

Before the old leader goes down, it is able to announce notification message. So 3 out 5 (including the old leader) elected the old leader to be a new leader for the next epoch. While, the old leader is being rebooted, 2 other machines are trying to connect to the old leader. So the quorum couldn't form until those 2 machines give up and move to the next round of leader election.

This is because Learner.connectToLeader() use a simple retry logic. The contract for this method is that it should never spend longer that initLimit trying to connect to the leader. In our outage, each sock.connect() is probably blocked for initLimit and it is called 5 times.
369113 No Perforce job exists for this issue. 3 369418
1 year, 17 weeks, 1 day ago 0|i1rmbr:
ZooKeeper ZOOKEEPER-1864

quorumVerifier is null when creating a QuorumPeerConfig from parsing a Properties object

Bug Resolved Major Fixed Michi Mutsuzaki some one some one 20/Jan/14 01:09   20/May/14 07:09 17/May/14 07:30   3.5.0 server   0 6   This bug was found when using ZK 3.5.0 with curator-test 2.3.0.
curator-test is building a QuorumPeerConfig from a Properties object and then when we try to run the quorum peer using that configuration, we get an NPE:
{noformat}
2014-01-19 21:58:39,768 [myid:] - ERROR [Thread-3:TestingZooKeeperServer$1@138] - From testing server (random state: false)
java.lang.NullPointerException
at org.apache.zookeeper.server.quorum.QuorumPeer.setQuorumVerifier(QuorumPeer.java:1320)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:156)
at org.apache.curator.test.TestingZooKeeperServer$1.run(TestingZooKeeperServer.java:134)
at java.lang.Thread.run(Thread.java:722)
{noformat}
The reason that this happens is because QuorumPeerConfig:parseProperties only peforms a subset of what 'QuorumPeerConfig:parse(String path)' does. The exact additional task performed that we need in parseProperties is the dynamic config backwards compatibility check:
{noformat}
// backward compatibility - dynamic configuration in the same file as static configuration params
// see writeDynamicConfig() - we change the config file to new format if reconfig happens
if (dynamicConfigFileStr == null) {
configBackwardCompatibilityMode = true;
configFileStr = path;................
parseDynamicConfig(cfg, electionAlg, true);
checkValidity();................
}
{noformat}
368737 No Perforce job exists for this issue. 2 369041
5 years, 44 weeks, 2 days ago 0|i1rk07:
ZooKeeper ZOOKEEPER-1863

Race condition in commit processor leading to out of order request completion, xid mismatch on client.

Bug Resolved Blocker Fixed Dutch T. Meyer Dutch T. Meyer Dutch T. Meyer 15/Jan/14 13:23   01/Aug/16 18:27 15/Jul/14 18:40 3.5.0 3.5.0 server   1 13   In CommitProcessor.java processor, if we are at the primary request handler on line 167:
{noformat}
while (!stopped && !isWaitingForCommit() &&
!isProcessingCommit() &&
(request = queuedRequests.poll()) != null) {
if (needCommit(request)) {
nextPending.set(request);
} else {
sendToNextProcessor(request);
}
}
{noformat}

A request can be handled in this block and be quickly processed and completed on another thread. If queuedRequests is empty, we then exit the block. Next, before this thread makes any more progress, we can get 2 more requests, one get_children(say), and a sync placed on queuedRequests for the processor. Then, if we are very unlucky, the sync request can complete and this object's commit() routine is called (from FollowerZookeeperServer), which places the sync request on the previously empty committedRequests queue. At that point, this thread continues.

We reach line 182, which is a check on sync requests.
{noformat}
if (!stopped && !isProcessingRequest() &&
(request = committedRequests.poll()) != null) {
{noformat}

Here we are not processing any requests, because the original request has completed. We haven't dequeued either the read or the sync request in this processor. Next, the poll above will pull the sync request off the queue, and in the following block, the sync will get forwarded to the next processor.

This is a problem because the read request hasn't been forwarded yet, so requests are now out of order.

I've been able to reproduce this bug reliably by injecting a Thread.sleep(5000) between the two blocks above to make the race condition far more likely, then in a client program.

{noformat}
zoo_aget_children(zh, "/", 0, getchildren_cb, NULL);
//Wait long enough for queuedRequests to drain
sleep(1);
zoo_aget_children(zh, "/", 0, getchildren_cb, &th_ctx[0]);
zoo_async(zh, "/", sync_cb, &th_ctx[0]);
{noformat}

When this bug is triggered, 3 things can happen:
1) Clients will see requests complete out of order and fail on xid mismatches.
2) Kazoo in particular doesn't handle this runtime exception well, and can orphan outstanding requests.
3) I've seen zookeeper servers deadlock, likely because the commit cannot be completed, which can wedge the commit processor.
368022 No Perforce job exists for this issue. 8 368329
3 years, 33 weeks, 3 days ago 0|i1rfn3:
ZooKeeper ZOOKEEPER-1862

ServerCnxnTest.testServerCnxnExpiry is intermittently failing

Bug Resolved Major Fixed Rakesh Radhakrishnan Rakesh Radhakrishnan Rakesh Radhakrishnan 12/Jan/14 22:50   16/Mar/14 13:53 14/Mar/14 19:11 3.5.0 3.5.0 tests   0 5   ServerCnxnTest#testServerCnxnExpiry test case is failing in the trunk build with the following exception
{code}
[junit] 2014-01-11 10:13:07,696 [myid:] - INFO [NIOServerCxnFactory.AcceptThread:0.0.0.0/0.0.0.0:11221:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:63930
[junit] 2014-01-11 10:13:09,000 [myid:] - INFO [ConnnectionExpirer:NIOServerCnxn@1006] - Closed socket connection for client /127.0.0.1:63930 (no session established for client)
[junit] 2014-01-11 10:13:10,697 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@62] - TEST METHOD FAILED testServerCnxnExpiry
[junit] java.net.SocketException: Software caused connection abort: recv failed
[junit] at java.net.SocketInputStream.socketRead0(Native Method)
[junit] at java.net.SocketInputStream.read(SocketInputStream.java:150)
[junit] at java.net.SocketInputStream.read(SocketInputStream.java:121)
[junit] at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:283)
[junit] at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:325)
[junit] at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:177)
[junit] at java.io.InputStreamReader.read(InputStreamReader.java:184)
[junit] at java.io.BufferedReader.fill(BufferedReader.java:154)
[junit] at java.io.BufferedReader.readLine(BufferedReader.java:317)
[junit] at java.io.BufferedReader.readLine(BufferedReader.java:382)
[junit] at org.apache.zookeeper.test.ServerCnxnTest.send4LetterWord(ServerCnxnTest.java:105)
[junit] at org.apache.zookeeper.test.ServerCnxnTest.sendRequest(ServerCnxnTest.java:77)
[junit] at org.apache.zookeeper.test.ServerCnxnTest.testServerCnxnExpiry(ServerCnxnTest.java:64)
[junit] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[junit] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
[junit] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[junit] at java.lang.reflect.Method.invoke(Method.java:601)
[junit] at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
[junit] at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
[junit] at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
[junit] at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
[junit] at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:52)
[junit] at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
[junit] at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
[junit] at org.junit.rules.TestWatchman$1.evaluate(TestWatchman.java:52)
[junit] at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
[junit] at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:69)
[junit] at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:48)
[junit] at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
[junit] at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
[junit] at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
[junit] at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
[junit] at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
[junit] at org.junit.runners.ParentRunner.run(ParentRunner.java:292)
[junit] at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:39)
[junit] at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:518)
[junit] at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1052)
[junit] at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:906)
{code}

When analyzing the possible cause of the failure is:
During connection expiry server will close the socket channel connection. After the socket closure, when the client tries to read a line of text will throw java.net.SocketException.

In the failure scenario the testcase has established a socket connection and entering into the sleep. In the meantime the server side expiration would happen and closing the socket channel. Assume after the socket closure the testcase is trying to read the text using the previously established socket and is resulting in SocketException. There is a race between the reading the socket in the client side and socket closure in server side.
{code}
NIOServerCnxn#closeSock is closing the socket channel.
sock.socket().shutdownOutput();
sock.socket().shutdownInput();
sock.socket().close();
sock.close();
{code}
367513 No Perforce job exists for this issue. 2 367821
6 years, 1 week, 4 days ago 0|i1rcj3:
ZooKeeper ZOOKEEPER-1861

ConcurrentHashMap isn't used properly in QuorumCnxManager

Bug Resolved Minor Fixed Ted Yu Ted Yu Ted Yu 11/Jan/14 18:50   13/Feb/14 22:03 13/Feb/14 22:03 3.5.0 3.5.0     0 6   queueSendMap is a ConcurrentHashMap.
At line 210:
{code}
if (!queueSendMap.containsKey(sid)) {
queueSendMap.put(sid, new ArrayBlockingQueue<ByteBuffer>(
SEND_CAPACITY));
{code}
By the time control enters if block, there may be another concurrent put with same sid to the ConcurrentHashMap.
putIfAbsent() should be used.

Similar issue occurs at line 307 as well.
367438 No Perforce job exists for this issue. 3 367747
6 years, 5 weeks, 6 days ago 0|i1rc2n:
ZooKeeper ZOOKEEPER-1860

Async versions of reconfig don't actually throw KeeperException nor InterruptedException

Bug Resolved Major Fixed Raúl Gutiérrez Segalés Raúl Gutiérrez Segalés Raúl Gutiérrez Segalés 11/Jan/14 15:54   14/Jan/14 06:05 13/Jan/14 17:34 3.5.0 3.5.0 java client   0 5   This was caught by [~fournc], the async versions of reconfig in the Java client don't actually throw KeeperException nor InterruptedException. Since this is unreleased code (i.e.: for 3.5.0) I don't think there are issues with changing the API (that is, considering what exceptions are thrown part of the API). 367427 No Perforce job exists for this issue. 1 367736
6 years, 10 weeks, 2 days ago
Reviewed
0|i1rc07:
ZooKeeper ZOOKEEPER-1859

pwriter should be closed in NIOServerCnxn#checkFourLetterWord()

Bug Open Minor Unresolved Unassigned Ted Yu Ted Yu 11/Jan/14 13:20   12/Jul/18 06:32           0 4   {code}
final PrintWriter pwriter = new PrintWriter(
new BufferedWriter(new SendBufferWriter()));
...
} else if (len == telnetCloseCmd) {
cleanupWriterSocket(null);
return true;
}
{code}
pwriter should be closed in case of telnetCloseCmd
367422 No Perforce job exists for this issue. 0 367731
1 year, 36 weeks ago 0|i1rbz3:
ZooKeeper ZOOKEEPER-1858

ZOOKEEPER-1833 JMX checks - potential race conditions while stopping and starting server

Sub-task Closed Major Fixed Rakesh Radhakrishnan Rakesh Radhakrishnan Rakesh Radhakrishnan 03/Jan/14 13:55   24/Mar/17 13:20 24/Jan/14 19:19   3.4.6, 3.5.0     0 5   I've noticed one potential case, where previously created zkclient session immediately reconnected and publishing those beans while starting back the zkserver and affecting zk#startup jmx checks.

Say, before stopping the server, there is a zk client session 0x143576544c50000 exists. While starting back the server, there could be possibility of seeing the client sessions in jmx. Following is one such case. Please see below logs which has taken from build https://builds.apache.org/job/ZooKeeper-trunk-WinVS2008_java/642/

{code} [junit] 2014-01-03 09:18:12,809 [myid:] - INFO [main-SendThread(127.0.0.1:11222):ClientCnxn$SendThread@1228] - Session establishment complete on server 127.0.0.1/127.0.0.1:11222, sessionid = 0x143576544c50000, negotiated timeout = 30000
[junit] 2014-01-03 09:18:12,809 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11222:ZooKeeperServer@617] - Established session 0x143576544c50000 with negotiated timeout 30000 for client /127.0.0.1:55377{code}

{code} [junit] 2014-01-03 09:18:12,391 [myid:] - INFO [main:JMXEnv@135] - ensureOnly:[]
[junit] 2014-01-03 09:18:12,395 [myid:] - INFO [main:ClientBase@438] - STARTING server
[junit] 2014-01-03 09:18:12,395 [myid:] - INFO [main:ClientBase@359] - CREATING server instance 127.0.0.1:11222
[junit] 2014-01-03 09:18:12,395 [myid:] - INFO [main:NIOServerCnxnFactory@94] - binding to port 0.0.0.0/0.0.0.0:11222
[junit] 2014-01-03 09:18:12,395 [myid:] - INFO [main:ClientBase@334] - STARTING server instance 127.0.0.1:11222

[junit] 2014-01-03 09:18:19,030 [myid:] - INFO [main:JMXEnv@142] - unexpected:org.apache.ZooKeeperService:name0=StandaloneServer_port-1,name1=Connections,name2=127.0.0.1,name3=0x143576544c50000
[junit] 2014-01-03 09:18:19,030 [myid:] - INFO [main:JMXEnv@142] - unexpected:org.apache.ZooKeeperService:name0=StandaloneServer_port-1
[junit] 2014-01-03 09:18:19,030 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@62] - TEST METHOD FAILED testDefaultWatcherAutoResetWithChroot
[junit] junit.framework.AssertionFailedError: expected:<0> but was:<2>
[junit] at junit.framework.Assert.fail(Assert.java:47)
[junit] at junit.framework.Assert.failNotEquals(Assert.java:283)
[junit] at junit.framework.Assert.assertEquals(Assert.java:64)
[junit] at junit.framework.Assert.assertEquals(Assert.java:195)
[junit] at junit.framework.Assert.assertEquals(Assert.java:201)
[junit] at org.apache.zookeeper.test.JMXEnv.ensureOnly(JMXEnv.java:144)
[junit] at org.apache.zookeeper.test.ClientBase.startServer(ClientBase.java:443)
[junit] at org.apache.zookeeper.test.DisconnectedWatcherTest.testDefaultWatcherAutoResetWithChroot(DisconnectedWatcherTest.java:123)
[junit] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
{code}
test 366221 No Perforce job exists for this issue. 8 366532
6 years, 2 weeks ago 0|i1r4l3:
ZooKeeper ZOOKEEPER-1857

ZOOKEEPER-1833 PrepRequestProcessotTest doesn't shutdown ZooKeeper server

Sub-task Closed Major Fixed Germán Blanco Germán Blanco Germán Blanco 03/Jan/14 01:57   13/Mar/14 14:17 09/Jan/14 18:04 3.4.5, 3.5.0 3.4.6, 3.5.0 tests   0 5   366126 No Perforce job exists for this issue. 4 366437
6 years, 2 weeks ago 0|i1r3zz:
ZooKeeper ZOOKEEPER-1856

zookeeper C-client can fail to switch from a dead server in a 3+ server ensemble if the client only has a 2 server list.

Bug Patch Available Major Unresolved Michi Mutsuzaki Dutch T. Meyer Dutch T. Meyer 02/Jan/14 18:43   05/Feb/20 07:11     3.7.0, 3.5.8 c client   0 5   If a client has a 2 server list, and is currently connected to the last server in that list, and that server then goes offline, the addrvec_next() call handle_error() will push the client to the start of the list and terminate the connection.

Then, the zoo_cycle_next_server() call in zookeeper_interest will be called in response to the connection failure, and the client will cycle back to the failed server.

In this way, a client who has a list of only 2 servers can get stuck on the one failed server. This would only be an issue in an ensemble larger than 2 of course, because failing 1 out of 2 would lead to quorum loss anyway.

There are other harmonics possible if every other server in the list is failed, but this is simplest to reproduce in a 3 server ensemble where the client only knows about 2 servers, one of which then fails. There are probably some elegant fixes here, but I think the simplest is to add a flag to track whether a server has been accessed before, and if it hasn't, don't call zoo_cycle_next_server() at the top of the zookeeper_interest() function.
366070 No Perforce job exists for this issue. 1 366381
1 year, 45 weeks ago 0|i1r3nr:
ZooKeeper ZOOKEEPER-1855

calls to zoo_set_server() fail to flush outstanding request queue.

Bug Open Minor Unresolved Unassigned Dutch T. Meyer Dutch T. Meyer 02/Jan/14 18:07   05/Feb/20 07:16     3.7.0, 3.5.8 c client   0 3   If one calls zoo_set_servers to update with a new server list that does not contain the currently connected server, the client will disconnect. Fair enough, but any outstanding requests on the set_requests queue aren't completed, so the next completed request from the new server can fail with an out-of-order XID error.

The disconnect occurs in update_addrs(), when a reconfig is necessary, though it's not quite as easy as just calling cleanup_bufs there, because you could then race the call to dequeue_completion in zookeeper_process and pull NULL entries for a recently completed request

I don't have a patch for this right now, but I do have a simple repro I can post when time permits.
366063 No Perforce job exists for this issue. 0 366374
6 years, 11 weeks, 3 days ago 0|i1r3m7:
ZooKeeper ZOOKEEPER-1854

ZOOKEEPER-1833 ClientBase ZooKeeper server clean-up

Sub-task Closed Major Invalid Germán Blanco Germán Blanco Germán Blanco 30/Dec/13 04:47   13/Mar/14 14:17 30/Dec/13 06:46 3.4.5, 3.5.0 3.4.6, 3.5.0 tests   0 3   Windows 7, Java 1.7 The ClientBase utility for tests provides methods for creating a ZooKeeper server, however the close up methods don't seem to shutdown that ZooKeeper server. 365727 No Perforce job exists for this issue. 2 366034
6 years, 2 weeks ago 0|i1r1iv:
ZooKeeper ZOOKEEPER-1853

zkCli.sh can't issue a CREATE command containing spaces in the data

Bug Closed Minor Fixed Ryan Lamore sekine coulibaly sekine coulibaly 28/Dec/13 17:38   21/Jul/16 16:18 08/Nov/15 17:27 3.4.6, 3.5.0 3.4.7, 3.5.2, 3.6.0 java client   3 13   Execute the following command in zkCli.sh :

create /contacts/1 {"country":"CA","name":"De La Salle"}

The results is that only {"id":1,"fullname":"De is stored.
The expected result is to have the full JSON payload stored.

The CREATE command seems to be croped after the first space of the data payload. When issuing a create command, all arguments not being -s nor -e shall be treated as the actual data.

patch 365640 No Perforce job exists for this issue. 5 365947
4 years, 18 weeks, 3 days ago Allows spaces to be used for parameters in zkCli as long as they are in single or double quotes. ie: create /node1 "This will now work" 0|i1r0zj:
ZooKeeper ZOOKEEPER-1852

ZOOKEEPER-1833 ServerCnxnFactory instance is not properly cleanedup

Sub-task Closed Major Fixed Rakesh Radhakrishnan Rakesh Radhakrishnan Rakesh Radhakrishnan 26/Dec/13 05:20   13/Mar/14 14:17 02/Jan/14 19:48   3.4.6, 3.5.0 tests   0 6   ClientBase#createNewServerInstance() - Say the startup of the server fails, this will not initialize 'serverFactory' and will be null. When the flow comes to teardown/shutdown, it will bypass stopping of this server instance due to the following check. This will affect other test case verifications like, jmx check 'JMXEnv#ensureOnly'.

ClientBase#shutdownServerInstance
{code}
static void shutdownServerInstance(ServerCnxnFactory factory,
String hostPort)
{
if (factory != null) {
//...shutdown logic
}
{code}
365384 No Perforce job exists for this issue. 6 365686
6 years, 2 weeks ago 0|i1qz9b:
ZooKeeper ZOOKEEPER-1851

Follower and Observer Request Processors Do Not Forward create2 Requests

Bug Resolved Blocker Fixed Chris Chen Chris Chen Chris Chen 23/Dec/13 18:23   19/Jul/14 07:24 18/Jul/14 13:49 3.5.0 3.5.0 quorum   0 7   Recent changes to the Observer and Follower Request Processors switch on the request opcode, but create2 is left out. This leads to a condition where the create2 request is passed to the CommitProcessor, but the leader never gets the request, the CommitProcessor can't find a matching request, so the client gets disconnected.

Added tests as well.
patch 365202 No Perforce job exists for this issue. 5 365507
5 years, 35 weeks, 5 days ago 0|i1qy5j:
ZooKeeper ZOOKEEPER-1850

cppunit test testNonexistingHost in TestZookeeperInit is failing on Unbuntu

Bug Closed Trivial Won't Fix Germán Blanco Germán Blanco Germán Blanco 21/Dec/13 12:52   13/Mar/14 14:17 22/Dec/13 12:18 3.4.6, 3.5.0 3.4.6, 3.5.0 tests   0 3   Linux ubuntu 3.8.0-29-generic #42~precise1-Ubuntu This is the error:
TestZookeeperInit.cc:241: Assertion: assertion failed [Expression: zh==0]
364866 No Perforce job exists for this issue. 1 365166
6 years, 2 weeks ago 0|i1qw1r:
ZooKeeper ZOOKEEPER-1849

ZOOKEEPER-1833 Need to properly tear down tests in various cases

Sub-task Closed Blocker Fixed Germán Blanco Germán Blanco Germán Blanco 17/Dec/13 22:12   13/Mar/14 14:16 21/Dec/13 16:06 3.4.5, 3.5.0 3.4.6, 3.5.0 tests   0 3   364377 No Perforce job exists for this issue. 3 364677
6 years, 2 weeks ago 0|i1qt1j:
ZooKeeper ZOOKEEPER-1848

[WINDOWS] Java NIO socket channels does not work with Windows ipv6 on JDK6

Bug Open Major Unresolved Enis Soztutar Enis Soztutar Enis Soztutar 17/Dec/13 20:17   05/Feb/20 07:17     3.7.0, 3.5.8     0 3   ZK uses Java NIO to create ServerSorcket's from ServerSocketChannels. Under windows, the ipv4 and ipv6 is implemented independently, and Java seems that it cannot reuse the same socket channel for both ipv4 and ipv6 sockets. We are getting "java.net.SocketException: Address family not supported by protocol
family" exceptions. When, ZK client resolves "localhost", it gets both v4 127.0.0.1 and v6 ::1 address, but the socket channel cannot bind to both v4 and v6.
The problem is reported as:
http://bugs.sun.com/view_bug.do?bug_id=6230761
http://stackoverflow.com/questions/1357091/binding-an-ipv6-server-socket-on-windows
Although the JDK bug is reported as resolved, I have tested with jdk1.6.0_33 without any success. Although JDK7 seems to have fixed this problem.

See HBASE-6825 for reference.
364365 No Perforce job exists for this issue. 2 364665
5 years, 50 weeks ago 0|i1qsyv:
ZooKeeper ZOOKEEPER-1847

Normalize line endings in repository

Bug Resolved Major Duplicate Enis Soztutar Enis Soztutar Enis Soztutar 17/Dec/13 18:05   05/Feb/14 16:31 05/Feb/14 16:31   3.5.0     0 3   It is good practice to have all the code in the repository use the same line endings (LF) so that patches can be applied normally. We can add a gitattributes file so that checked out code can still have platform dependent line endings.

More readings:
https://help.github.com/articles/dealing-with-line-endings
http://stackoverflow.com/questions/170961/whats-the-best-crlf-handling-strategy-with-git
364348 No Perforce job exists for this issue. 1 364648
6 years, 7 weeks, 1 day ago 0|i1qsv3:
ZooKeeper ZOOKEEPER-1846

Cached InetSocketAddresses prevent proper dynamic DNS resolution

Bug Resolved Minor Duplicate Unassigned Benjamin Jaton Benjamin Jaton 16/Dec/13 20:18   30/Jan/19 07:50 16/Jan/17 01:40 3.4.6   quorum   1 12 0 600   The class QuorumPeer maintains a Map<Long, QuorumServer> quorumPeers.
Each QuorumServer is created with an instance of InetSocketAddress electionAddr, and holds it forever.

I believe this is why the ZooKeeper servers can't resolve each other dynamically: If a ZooKeeper in the ensemble cannot be resolved at startup, it will never be resolved (until restart of the JVM), constantly failing with an UnknownHostException, even when the node is back up and reachable.

I would suggest to recreate an InetSocketAddress every time we retry the connection.
100% 100% 600 0 patch, pull-request-available 364152 No Perforce job exists for this issue. 4 364452
3 years, 9 weeks, 3 days ago Forces the re-resolve, on error, of the Peers' Hostname to IP address, which is an issue in virtual/cloud environments where IPs are assigned dynamically upon every container startup.
If the Hostname is unresolvable or the connection fails (IP change), this DNS refresh process is immediately triggered.
0|i1qrnr:
ZooKeeper ZOOKEEPER-1845

FLETest.testLE fails on windows

Bug Closed Major Duplicate Michi Mutsuzaki Michi Mutsuzaki Michi Mutsuzaki 16/Dec/13 19:13   13/Mar/14 14:17 17/Dec/13 17:35 3.4.5 3.4.6 quorum, tests   0 3   windows 7 x64 java 1.7.0_45 This test waits for the leader election to settle, but it is possible that 3 follower threads join before the leader thread joins. We should wait for the leader thread to join in a loop for some time.

{noformat}
Leader hasn't joined: 5
junit.framework.AssertionFailedError: Leader hasn't joined: 5
at org.apache.zookeeper.test.FLETest.testLE(FLETest.java:313)
at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:52)
{noformat}
364146 No Perforce job exists for this issue. 1 364446
6 years, 2 weeks ago 0|i1qrmf:
ZooKeeper ZOOKEEPER-1844

TruncateTest fails on windows

Bug Closed Critical Fixed Rakesh Radhakrishnan Michi Mutsuzaki Michi Mutsuzaki 16/Dec/13 18:58   13/Mar/14 14:17 12/Feb/14 11:01 3.4.5, 3.5.0 3.4.6, 3.5.0 server   0 5   windows TruncateTest has been failing consistently on windows:

https://builds.apache.org/job/ZooKeeper-trunk-WinVS2008_java/627/testReport/junit/org.apache.zookeeper.test/TruncateTest/testTruncate/
364141 No Perforce job exists for this issue. 7 364441
6 years, 2 weeks ago 0|i1qrlb:
ZooKeeper ZOOKEEPER-1843

Oddity in ByteBufferInputStream skip

Bug Resolved Minor Fixed Bill Havanki Justin SB Justin SB 16/Dec/13 18:09   28/Apr/14 07:05 26/Apr/14 17:17   3.5.0     0 5   I was reading ByteBufferInputStream.java; here is the skip method:

public long skip(long n) throws IOException {
long newPos = bb.position() + n;
if (newPos > bb.remaining()) {
n = bb.remaining();
}
bb.position(bb.position() + (int) n);
return n;
}

The first two lines look wrong; we compare a "point" (position) to a "distance" (remaining). I think the test should be if (newPos >= bb.limit()).

Or more simply:
public long skip(long n) throws IOException {
int remaining = buffer.remaining();
if (n > remaining) {
n = remaining;
}
buffer.position(buffer.position() + (int) n);
return n;
}
364139 No Perforce job exists for this issue. 2 364439
5 years, 47 weeks, 3 days ago 0|i1qrkv:
ZooKeeper ZOOKEEPER-1842

Election listening thread not shutdown for leaders or followers

Bug Open Major Unresolved Unassigned Jeremy Stribling Jeremy Stribling 16/Dec/13 15:35   16/Dec/13 16:10       quorum   0 2   Linux, ZK trunk When I was testing the patch for https://issues.apache.org/jira/browse/ZOOKEEPER-1691, the test included with that patch was failing for me. The problem happened when the tests shuts down some followers and then attempts to bring them back up:

{quote}
2013-12-13 17:31:03,976 [myid:1] - INFO [QuorumPeer[myid=1]/127.0.0.1:11227:Follower@194] - shutdown called
java.lang.Exception: shutdown Follower
at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:194)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:971)
...
2013-12-13 17:31:03,992 [myid:1] - INFO [QuorumPeerListener:QuorumCnxManager$Listener@544] - My election bind port: localhost/127.0.0.1:11229
2013-12-13 17:31:03,992 [myid:1] - ERROR [localhost/127.0.0.1:11229:QuorumCnxManager$Listener@557] - Exception while listening
java.net.BindException: Address already in use
at java.net.PlainSocketImpl.socketBind(Native Method)
at java.net.AbstractPlainSocketImpl.bind(AbstractPlainSocketImpl.java:376)
at java.net.ServerSocket.bind(ServerSocket.java:376)
at java.net.ServerSocket.bind(ServerSocket.java:330)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$Listener.run(QuorumCnxManager.java:546)
{quote}

The problem appears to be that the when follower.shutdown() is called in QuorumPeer.run(), the election algorithm is never shut down, so when the node restarts it can't bind back to the same port.

I will upload a patch that calls shutdown() for both the leader and the follower in this case, but I'm not positive its the right place or fix for this issue, so feedback would be appreciated.
364115 No Perforce job exists for this issue. 1 364415
6 years, 14 weeks, 3 days ago 0|i1qrfj:
ZooKeeper ZOOKEEPER-1841

ZOOKEEPER-1833 problem in QuorumTest

Sub-task Closed Major Fixed Germán Blanco Germán Blanco Germán Blanco 16/Dec/13 09:32   13/Mar/14 14:16 18/Dec/13 08:06 3.4.5 3.4.6 tests   0 2   Windows, Java 1.7 QuorumTest.testNoLogBeforeLeaderEstablishment fails with Assertion: "NOt following" 364042 No Perforce job exists for this issue. 5 364342
6 years, 2 weeks ago 0|i1qqzb:
ZooKeeper ZOOKEEPER-1840

Server tries to connect to itself during dynamic reconfig

Bug Resolved Minor Fixed Alexander Shraer Bruno Freudensprung Bruno Freudensprung 14/Dec/13 06:53   16/Apr/14 07:07 16/Apr/14 03:07 3.5.0   quorum   0 5   Submitted this bug on a suggestion of Alexander Shraer (see https://issues.apache.org/jira/browse/ZOOKEEPER-1691)

How to reproduce:

== Server 1 zoo.cfg:
standaloneEnabled=false
dynamicConfigFile=<path to>/confdyn1/zoo.cfg.dynamic

== Server 1 zoo.cfg.dynamic:
server.1=localhost:2888:3888:participant;localhost:2181

== Server 2 zoo.cfg:
standaloneEnabled=false
dynamicConfigFile=<path to>/confdyn2/zoo.cfg.dynamic

== Server 2 zoo.cfg.dynamic (it is "aware" of the server 1, as mentioned in the Dynamic Reconfiguration - User Manual
that I should have read more carefully yesterday):
server.1=localhost:2888:3888:participant;localhost:2181
server.2=localhost:2889:3889:participant;localhost:2182

Start server 1
Start server 2

== use client 1 to issue a reconfig command on server 1:
[zk: localhost:2181(CONNECTED) 1] reconfig -add server.2=localhost:2889:3889:participant;localhost:2182
Committed new configuration:
server.1=localhost:2888:3888:participant;localhost:2181
server.2=localhost:2889:3889:participant;localhost:2182
version=100000003

There are strange stack traces in both server consoles.

Server 1:
2013-12-12 22:31:40,888 [myid:1] - WARN [ProcessThread(sid:1 cport:-1)::QuorumCnxManager@390] - Cannot open channel to 2 at election address localhost/127.0.0.1:3889
java.net.ConnectException: Connection refused: connect
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)
at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
at java.net.Socket.connect(Socket.java:529)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:375)
at org.apache.zookeeper.server.quorum.QuorumPeer.connectNewPeers(QuorumPeer.java:1252)
at org.apache.zookeeper.server.quorum.QuorumPeer.setLastSeenQuorumVerifier(QuorumPeer.java:1272)
at org.apache.zookeeper.server.quorum.Leader.propose(Leader.java:1071)
at org.apache.zookeeper.server.quorum.ProposalRequestProcessor.processRequest(ProposalRequestProcessor.java:78)
at org.apache.zookeeper.server.PrepRequestProcessor.pRequest(PrepRequestProcessor.java:864)
at org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:144)
2013-12-12 22:31:41,919 [myid:1] - WARN [LearnerHandler-/127.0.0.1:52301:QuorumPeer@1259] - Restarting Leader Election
2013-12-12 22:31:41,920 [myid:1] - INFO [localhost/127.0.0.1:3888:QuorumCnxManager$Listener@571] - Leaving listener
2013-12-12 22:31:41,920 [myid:1] - INFO [QuorumPeerListener:QuorumCnxManager$Listener@544] - My election bind port: localhost/127.0.0.1:3888
2013-12-12 22:31:44,438 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection$Messenger$WorkerReceiver@410] - WorkerReceiver is down
2013-12-12 22:31:44,439 [myid:1] - INFO [WorkerSender[myid=1]:FastLeaderElection$Messenger$WorkerSender@442] - WorkerSender is down

Server 2:
2013-12-12 22:31:41,894 [myid:2] - WARN [QuorumPeer[myid=2]/127.0.0.1:2182:QuorumCnxManager@390] - Cannot open channel to 2 at election address localhost/127.0.0.1:3889
java.net.ConnectException: Connection refused: connect
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)
at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
at java.net.Socket.connect(Socket.java:529)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:375)
at org.apache.zookeeper.server.quorum.QuorumPeer.connectNewPeers(QuorumPeer.java:1252)
at org.apache.zookeeper.server.quorum.QuorumPeer.setLastSeenQuorumVerifier(QuorumPeer.java:1272)
at org.apache.zookeeper.server.quorum.Follower.processPacket(Follower.java:131)
at org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:89)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:967)
2013-12-12 22:31:41,923 [myid:2] - WARN [QuorumPeer[myid=2]/127.0.0.1:2182:QuorumPeer@1259] - Restarting Leader Election
2013-12-12 22:31:41,924 [myid:2] - INFO [QuorumPeerListener:QuorumCnxManager$Listener@544] - My election bind port: localhost/127.0.0.1:3889
363737 No Perforce job exists for this issue. 1 364043
5 years, 49 weeks, 1 day ago 0|i1qp53:
ZooKeeper ZOOKEEPER-1839

Deadlock in NettyServerCnxn

Bug Closed Critical Fixed Rakesh Radhakrishnan Rakesh Radhakrishnan Rakesh Radhakrishnan 13/Dec/13 09:25   13/Mar/14 14:16 15/Dec/13 23:21 3.4.5 3.4.6, 3.5.0 server   0 7   Deadlock found during NettyServerCnxn closure. Please see the attached threaddump. 363540 No Perforce job exists for this issue. 3 363846
6 years, 2 weeks ago 0|i1qnxj:
ZooKeeper ZOOKEEPER-1838

ZOOKEEPER-1833 ZooKeeper shutdown hangs indefinitely at NioServerSocketChannelFactory.releaseExternalResources

Sub-task Closed Major Duplicate Rakesh Radhakrishnan Rakesh Radhakrishnan Rakesh Radhakrishnan 13/Dec/13 07:18   13/Mar/14 14:17 13/Dec/13 14:24   3.4.6 server   0 5   Zookeeper shutdown hangs when releasing external resources. This has been observed when executing NioNettySuiteTest.
{code}
"main" prio=6 tid=0x01498400 nid=0x2328 waiting on condition [0x0158e000..0x0158fe28]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x22f58918> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)
at java.util.concurrent.ThreadPoolExecutor.awaitTermination(ThreadPoolExecutor.java:1244)
at org.jboss.netty.util.internal.ExecutorUtil.terminate(ExecutorUtil.java:87)
at org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.releaseExternalResources(NioServerSocketChannelFactory.java:146)
at org.jboss.netty.bootstrap.Bootstrap.releaseExternalResources(Bootstrap.java:324)
at org.apache.zookeeper.server.NettyServerCnxnFactory.shutdown(NettyServerCnxnFactory.java:345)
at org.apache.zookeeper.test.ClientBase.shutdownServerInstance(ClientBase.java:355)
at org.apache.zookeeper.test.ClientBase.stopServer(ClientBase.java:422)
{code}
363530 No Perforce job exists for this issue. 2 363836
6 years, 2 weeks ago 0|i1qnvb:
ZooKeeper ZOOKEEPER-1837

ZOOKEEPER-1833 Fix JMXEnv checks (potential race conditions)

Sub-task Closed Major Fixed Germán Blanco Germán Blanco Germán Blanco 13/Dec/13 02:40   13/Mar/14 14:16 16/Jan/14 18:01 3.4.5, 3.5.0 3.4.6, 3.5.0 tests   0 5   Windows 8 The following failures in ZooKeeper-3.4-WinVS2008_java and ZooKeeper-trunk-WinVS2008_java require fixing:
[junit] junit.framework.AssertionFailedError: expected:<0> but was:<1>
[junit] at junit.framework.Assert.fail(Assert.java:47)
[junit] at junit.framework.Assert.failNotEquals(Assert.java:283)
[junit] at junit.framework.Assert.assertEquals(Assert.java:64)
[junit] at junit.framework.Assert.assertEquals(Assert.java:195)
[junit] at junit.framework.Assert.assertEquals(Assert.java:201)
[junit] at org.apache.zookeeper.test.JMXEnv.ensureOnly(JMXEnv.java:138)
[junit] at org.apache.zookeeper.test.ClientBase.startServer(ClientBase.java:417)
[junit] at org.apache.zookeeper.test.ZooKeeperQuotaTest.testQuota(ZooKeeperQuotaTest.java:80)
[junit] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[junit] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
[junit] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
[junit] at java.lang.reflect.Method.invoke(Method.java:597)
[junit] at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
[junit] at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
[junit] at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
[junit] at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)


[junit] junit.framework.AssertionFailedError: expected:<0> but was:<1>
[junit] at junit.framework.Assert.fail(Assert.java:47)
[junit] at junit.framework.Assert.failNotEquals(Assert.java:283)
[junit] at junit.framework.Assert.assertEquals(Assert.java:64)
[junit] at junit.framework.Assert.assertEquals(Assert.java:195)
[junit] at junit.framework.Assert.assertEquals(Assert.java:201)
[junit] at org.apache.zookeeper.test.JMXEnv.ensureOnly(JMXEnv.java:138)
[junit] at org.apache.zookeeper.test.ClientBase.startServer(ClientBase.java:417)
[junit] at org.apache.zookeeper.test.ZooKeeperQuotaTest.testQuota(ZooKeeperQuotaTest.java:80)
[junit] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[junit] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
[junit] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
[junit] at java.lang.reflect.Method.invoke(Method.java:597)
[junit] at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
[junit] at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
[junit] at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
[junit] at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
[junit] at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:52)
[

junit.framework.AssertionFailedError: expected [0x142e5f027b50001] expected:<1> but was:<0>
at org.apache.zookeeper.test.JMXEnv.ensureAll(JMXEnv.java:115)
at org.apache.zookeeper.test.ClientBase.createClient(ClientBase.java:197)
at org.apache.zookeeper.test.ClientBase.createClient(ClientBase.java:171)
at org.apache.zookeeper.test.ClientBase.createClient(ClientBase.java:156)
at org.apache.zookeeper.test.ClientBase.createClient(ClientBase.java:149)
at org.apache.zookeeper.ZooKeeperTest.testDeleteRecursive(ZooKeeperTest.java:45)
at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:52)
363493 No Perforce job exists for this issue. 12 363799
6 years, 2 weeks ago 0|i1qnn3:
ZooKeeper ZOOKEEPER-1836

addrvec_next() fails to set next parameter if addrvec_hasnext() returns false

Bug Resolved Trivial Fixed Dutch T. Meyer Dutch T. Meyer Dutch T. Meyer 12/Dec/13 19:50   20/May/14 07:09 15/May/14 16:40   3.5.0 c client   0 5   There is a relatively innocuous but useless pointer assignment in
addrvec_next():

195 void addrvec_next(addrvec_t *avec, struct sockaddr_storage *next)
....
203 if (!addrvec_hasnext(avec))
204 {
205 next = NULL;
206 return;

That assignment on (205) has no point, as next is a local variable lost upon function return. Likely this should be a memset to zero out the actual parameter.
363459 No Perforce job exists for this issue. 2 363765
5 years, 44 weeks, 2 days ago 0|i1qnfr:
ZooKeeper ZOOKEEPER-1835

dynamic configuration file renaming fails on Windows

Bug Resolved Major Fixed Bruno Freudensprung Bruno Freudensprung Bruno Freudensprung 12/Dec/13 16:05   04/Jul/14 07:11 03/Jul/14 22:25 3.5.0 3.5.0 quorum   0 7   Windows 7 64-bit, Oracle Java 1.6.0_32-b05 On Windows, reconfig fails to rename the tmp dynamic config file to the real dynamic config filename.
Javadoc of java.io.File.renameTo says the behavior is highly plateform dependent, so I guess this should not be a big surprise.
The problem occurs in src/java/main/org/apache/zookeeper/server/quorum/QuorumPeerConfig.java that could be modified like this:
+ curFile.delete();
if (!tmpFile.renameTo(curFile)) {
+ configFile.delete();
if (!tmpFile.renameTo(configFile)) {
As suggested by Alex in https://issues.apache.org/jira/browse/ZOOKEEPER-1691 (btw there is more information about my test scenario over there) it is a bit "scary" to delete the current configuration file.
363424 No Perforce job exists for this issue. 6 363730
5 years, 37 weeks, 6 days ago Patch described in this comment below:
https://issues.apache.org/jira/browse/ZOOKEEPER-1835?focusedCommentId=13848420&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13848420
0|i1qn7z:
ZooKeeper ZOOKEEPER-1834

ZOOKEEPER-1833 Catch IOException in FileTxnLog

Sub-task Closed Major Fixed Flavio Paiva Junqueira Flavio Paiva Junqueira Flavio Paiva Junqueira 10/Dec/13 17:12   13/Mar/14 14:16 10/Dec/13 19:55   3.4.6, 3.5.0     0 3   Upon an IOException in FileTxnLog#next(), the log file open remains open, which causes test cases at least in BufferSizeTest to fail. We need to add a catch block. 362998 No Perforce job exists for this issue. 4 363304
6 years, 2 weeks ago 0|i1qklr:
ZooKeeper ZOOKEEPER-1833

fix windows build

Bug Resolved Blocker Fixed Michi Mutsuzaki Michi Mutsuzaki Michi Mutsuzaki 06/Dec/13 20:49   03/Nov/15 09:46 03/Nov/15 09:46 3.4.5 3.4.7     1 7   ZOOKEEPER-1414, ZOOKEEPER-1459, ZOOKEEPER-1834, ZOOKEEPER-1837, ZOOKEEPER-1838, ZOOKEEPER-1841, ZOOKEEPER-1849, ZOOKEEPER-1852, ZOOKEEPER-1854, ZOOKEEPER-1857, ZOOKEEPER-1858, ZOOKEEPER-1866, ZOOKEEPER-1867, ZOOKEEPER-1868, ZOOKEEPER-1872, ZOOKEEPER-1873, ZOOKEEPER-1874, ZOOKEEPER-1904, ZOOKEEPER-1905, ZOOKEEPER-2047 A bunch of 3.4 tests are failing on windows.

{noformat}
[junit] 2013-12-06 08:40:59,692 [myid:] - INFO [main:ZKTestCase$1@65] - FAILED testEarlyLeaderAbandonment
[junit] 2013-12-06 08:41:10,472 [myid:] - INFO [main:ZKTestCase$1@65] - FAILED testHighestZxidJoinLate
[junit] 2013-12-06 08:45:31,085 [myid:] - INFO [main:ZKTestCase$1@65] - FAILED testUpdatingEpoch
[junit] 2013-12-06 08:55:34,630 [myid:] - INFO [main:ZKTestCase$1@65] - FAILED testObserversHammer
[junit] 2013-12-06 08:55:59,889 [myid:] - INFO [main:ZKTestCase$1@65] - FAILED testAsyncExistsFailure_NoNode
[junit] 2013-12-06 08:56:00,571 [myid:] - INFO [main:ZKTestCase$1@65] - FAILED testAsyncGetACL
[junit] 2013-12-06 08:56:02,626 [myid:] - INFO [main:ZKTestCase$1@65] - FAILED testAsyncGetChildrenEmpty
[junit] 2013-12-06 08:56:03,491 [myid:] - INFO [main:ZKTestCase$1@65] - FAILED testAsyncGetChildrenSingle
[junit] 2013-12-06 08:56:11,276 [myid:] - INFO [main:ZKTestCase$1@65] - FAILED testAsyncGetChildrenTwo
[junit] 2013-12-06 08:56:13,878 [myid:] - INFO [main:ZKTestCase$1@65] - FAILED testAsyncGetChildrenFailure_NoNode
[junit] 2013-12-06 08:56:16,294 [myid:] - INFO [main:ZKTestCase$1@65] - FAILED testAsyncGetChildren2Empty
[junit] 2013-12-06 08:56:18,622 [myid:] - INFO [main:ZKTestCase$1@65] - FAILED testAsyncGetChildren2Single
[junit] 2013-12-06 08:56:21,224 [myid:] - INFO [main:ZKTestCase$1@65] - FAILED testAsyncGetChildren2Two
[junit] 2013-12-06 08:56:23,738 [myid:] - INFO [main:ZKTestCase$1@65] - FAILED testAsyncGetChildren2Failure_NoNode
[junit] 2013-12-06 08:56:26,058 [myid:] - INFO [main:ZKTestCase$1@65] - FAILED testAsyncGetData
[junit] 2013-12-06 08:56:28,482 [myid:] - INFO [main:ZKTestCase$1@65] - FAILED testAsyncGetDataFailure_NoNode
[junit] 2013-12-06 08:57:35,527 [myid:] - INFO [main:ZKTestCase$1@65] - FAILED testStartupFailureCreate
[junit] 2013-12-06 08:57:38,645 [myid:] - INFO [main:ZKTestCase$1@65] - FAILED testStartupFailureSet
[junit] 2013-12-06 08:57:41,261 [myid:] - INFO [main:ZKTestCase$1@65] - FAILED testStartupFailureSnapshot
[junit] 2013-12-06 08:59:22,222 [myid:] - INFO [main:ZKTestCase$1@65] - FAILED testClientWithWatcherObj
[junit] 2013-12-06 09:00:05,592 [myid:] - INFO [main:ZKTestCase$1@65] - FAILED testClientCleanup
[junit] 2013-12-06 09:01:24,113 [myid:] - INFO [main:ZKTestCase$1@65] - FAILED testBindByAddress
[junit] 2013-12-06 09:02:14,123 [myid:] - INFO [main:ZKTestCase$1@65] - FAILED testClientwithoutWatcherObj
[junit] 2013-12-06 09:05:56,461 [myid:] - INFO [main:ZKTestCase$1@65] - FAILED testZeroWeightQuorum
[junit] 2013-12-06 09:08:18,747 [myid:] - INFO [main:ZKTestCase$1@65] - FAILED testResyncByDiffAfterFollowerCrashes
[junit] 2013-12-06 09:09:42,271 [myid:] - INFO [main:ZKTestCase$1@65] - FAILED testFourLetterWords
[junit] 2013-12-06 09:14:03,770 [myid:] - INFO [main:ZKTestCase$1@65] - FAILED testLE
[junit] 2013-12-06 09:46:30,002 [myid:] - INFO [main:ZKTestCase$1@65] - FAILED testHierarchicalQuorum
[junit] 2013-12-06 09:50:26,912 [myid:] - INFO [main:ZKTestCase$1@65] - FAILED testHammerBasic
[junit] 2013-12-06 09:51:07,604 [myid:] - INFO [main:ZKTestCase$1@65] - FAILED testQuotaWithQuorum
[junit] 2013-12-06 09:52:41,515 [myid:] - INFO [main:ZKTestCase$1@65] - FAILED testNull
[junit] 2013-12-06 09:53:22,648 [myid:] - INFO [main:ZKTestCase$1@65] - FAILED testDeleteWithChildren
[junit] 2013-12-06 09:56:49,061 [myid:] - INFO [main:ZKTestCase$1@65] - FAILED testClientwithoutWatcherObj
[junit] 2013-12-06 09:58:27,705 [myid:] - INFO [main:ZKTestCase$1@65] - FAILED testGetView
[junit] 2013-12-06 09:59:07,856 [myid:] - INFO [main:ZKTestCase$1@65] - FAILED testViewContains
[junit] 2013-12-06 10:01:31,418 [myid:] - INFO [main:ZKTestCase$1@65] - FAILED testSessionMoved
[junit] 2013-12-06 10:04:50,542 [myid:] - INFO [main:ZKTestCase$1@65] - FAILED testMultiToFollower
[junit] 2013-12-06 10:07:55,361 [myid:] - INFO [main:ZKTestCase$1@65] - FAILED testBehindLeader
[junit] 2013-12-06 10:10:57,439 [myid:] - INFO [main:ZKTestCase$1@65] - FAILED testLateLogs
[junit] 2013-12-06 10:12:05,336 [myid:] - INFO [main:ZKTestCase$1@65] - FAILED testConnectionEvents
[junit] 2013-12-06 10:14:02,781 [myid:] - INFO [main:ZKTestCase$1@65] - FAILED testRecovery
[junit] 2013-12-06 10:14:37,220 [myid:] - INFO [main:ZKTestCase$1@65] - FAILED testFail
[junit] 2013-12-06 10:14:46,925 [myid:] - INFO [main:ZKTestCase$1@65] - FAILED testRestoreCommittedLog
[junit] 2013-12-06 10:15:30,109 [myid:] - INFO [main:ZKTestCase$1@65] - FAILED testAuth
[junit] 2013-12-06 10:16:09,256 [myid:] - INFO [main:ZKTestCase$1@65] - FAILED testAuth
[junit] 2013-12-06 10:16:44,586 [myid:] - INFO [main:ZKTestCase$1@65] - FAILED testAuth
[junit] 2013-12-06 10:17:19,222 [myid:] - INFO [main:ZKTestCase$1@65] - FAILED testBadSaslAuthNotifiesWatch
[junit] 2013-12-06 10:17:54,239 [myid:] - INFO [main:ZKTestCase$1@65] - FAILED testAuthFail
[junit] 2013-12-06 10:18:35,623 [myid:] - INFO [main:ZKTestCase$1@65] - FAILED testAuth
[junit] 2013-12-06 10:18:47,094 [myid:] - INFO [main:ZKTestCase$1@65] - FAILED testAuth
[junit] 2013-12-06 10:19:06,770 [myid:] - INFO [main:ZKTestCase$1@65] - FAILED testCreateAfterCloseShouldFail
[junit] 2013-12-06 10:20:23,884 [myid:] - INFO [main:ZKTestCase$1@65] - FAILED testBasic
{noformat}
362495 No Perforce job exists for this issue. 5 362789
4 years, 20 weeks, 2 days ago 0|i1qhh3:
ZooKeeper ZOOKEEPER-1832

Add count of connected clients to submitted ganglia metrics

Bug Open Minor Unresolved Unassigned Ben Hartshorne Ben Hartshorne 05/Dec/13 13:21   06/Dec/13 16:55           0 1   The ganglia zookeeper plugin does not report the number of connected clients, though this information is available from the 'stat' command. 362221 No Perforce job exists for this issue. 0 362516
6 years, 16 weeks ago 0|i1qfsf:
ZooKeeper ZOOKEEPER-1831

ZOOKEEPER-1829 Document remove watches details to the guide

Sub-task Resolved Major Fixed Rakesh Radhakrishnan Rakesh Radhakrishnan Rakesh Radhakrishnan 05/Dec/13 04:20   16/Dec/18 09:36 03/Apr/14 19:14   3.5.0 documentation   0 4   This JIRA is for documenting the details of removing the watches remove_watches 362120 No Perforce job exists for this issue. 2 362415
5 years, 50 weeks, 6 days ago 0|i1qf5z:
ZooKeeper ZOOKEEPER-1830

ZOOKEEPER-1829 Support command line shell for removing watches

Sub-task Resolved Critical Fixed Rakesh Radhakrishnan Rakesh Radhakrishnan Rakesh Radhakrishnan 04/Dec/13 23:57   16/Dec/18 09:36 03/Apr/14 19:04   3.5.0     0 4   This JIRA to discuss the command line shell for removing watches. Makes it easier to do ad-hoc testing. remove_watches 362090 No Perforce job exists for this issue. 2 362385
5 years, 50 weeks, 6 days ago 0|i1qezb:
ZooKeeper ZOOKEEPER-1829

Umbrella jira for removing watches that are no longer of interest

New Feature Resolved Critical Fixed Rakesh Radhakrishnan Rakesh Radhakrishnan Rakesh Radhakrishnan 04/Dec/13 23:57   16/Dec/18 09:35 04/Apr/14 13:52   3.5.0 java client, server   1 3   ZOOKEEPER-442, ZOOKEEPER-1830, ZOOKEEPER-1831 remove_watches 362089 No Perforce job exists for this issue. 0 362384
5 years, 50 weeks, 6 days ago 0|i1qez3:
ZooKeeper ZOOKEEPER-1828

Audit src/c/src/zookeeper.c for missing error checking & early returns

Bug Open Major Unresolved Unassigned Raúl Gutiérrez Segalés Raúl Gutiérrez Segalés 03/Dec/13 19:29   06/Dec/13 16:34       c client   0 1   After discussing the patch for ZOOKEEPER-1632 we came to realize that many methods in the c client don't honor the internal error checking that they do (i.e.: they don't return early on error). This is bad because you might return with an error but your call (in case of the async ones) might still have been called.

So at that point it's unclear how the caller should deal with resource deallocation.
361831 No Perforce job exists for this issue. 0 362128
6 years, 16 weeks, 2 days ago 0|i1qde7:
ZooKeeper ZOOKEEPER-1827

Reduce the number of mntr calls in zookeeper ganglia plugin

Improvement Open Major Unresolved Unassigned Nikhil Mulley Nikhil Mulley 03/Dec/13 16:08   03/Dec/13 16:08       contrib   0 1   The zookeeper ganglia plugin script makes number of mntr calls to the zookeeper node for each of the metric, which seems to be an overhead to the zookeeper. I think it could be improvised to make a single mntr call and collect the metric data and send all the metrics information collected from a single mntr command (that is run every 60s interval) to the ganglia.

The change and github pull request is at https://github.com/apache/zookeeper/pull/8

Please let me know if there are any other changes required here.

thanks,

Nikhil
361775 No Perforce job exists for this issue. 0 362072
6 years, 16 weeks, 2 days ago 0|i1qd1r:
ZooKeeper ZOOKEEPER-1826

zookeeper ganglia module fails to read the output of stat when too many clients are connected

Bug Open Minor Unresolved Unassigned Ben Hartshorne Ben Hartshorne 03/Dec/13 14:41   06/Dec/13 16:34           0 1   The ganglia zookeeper module uses a single 2048 byte socket.recv to get the response from the 'stat' command. When there are more than a few clients connected, the list of connected clients fills the buffer before the script gets to the actual metrics its trying to report.

This bug is fixed in https://github.com/maplebed/zookeeper-monitoring/tree/ben.fetch_more_data
361739 No Perforce job exists for this issue. 0 362036
6 years, 16 weeks, 2 days ago 0|i1qctr:
ZooKeeper ZOOKEEPER-1825

In ClientBase, if not set "build.test.dir" will throw IOException

Test Open Major Unresolved Unassigned chendihao chendihao 02/Dec/13 08:01   02/Dec/13 21:19       tests   0 1   {code}
static final File BASETEST = new File(System.getProperty("build.test.dir", "build"));
{code}

{code}
public static File createTmpDir() throws IOException {
return createTmpDir(BASETEST);
}
static File createTmpDir(File parentDir) throws IOException {
File tmpFile = File.createTempFile("test", ".junit", parentDir);
// don't delete tmpFile - this ensures we don't attempt to create
// a tmpDir with a duplicate name
File tmpDir = new File(tmpFile + ".dir");
Assert.assertFalse(tmpDir.exists()); // never true if tmpfile does it's job
Assert.assertTrue(tmpDir.mkdirs());

return tmpDir;
}
{code}

Because the default directory "build" may not exist, createTmpDir() will throws IOException and show "No such file or directory". I think replace "build" as "." is more reasonable.
361384 No Perforce job exists for this issue. 0 361683
6 years, 16 weeks, 3 days ago 0|i1qanr:
ZooKeeper ZOOKEEPER-1824

We need a MiniZooKeeperCluster for unit test

Test Open Major Unresolved Unassigned chendihao chendihao 02/Dec/13 03:10   16/Apr/14 04:57       tests   2 6   We are developing a timestamp server to offer a precise auto increasing timestamp for other processes. ZooKeeper is used to select the master and store the persistent max offered timestamp.

Now I'm not sure if the zookeeper servers work well no matter how much I damage the cluster randomly. So we need unit tests but ZooKeeper doesn't provides a single-process cluster for us. Should we implement the similar code like what HBase did in MiniZooKeeperCluster.java ?

ref https://issues.apache.org/jira/browse/HBASE-2218
361332 No Perforce job exists for this issue. 0 361631
5 years, 49 weeks, 1 day ago 0|i1qac7:
ZooKeeper ZOOKEEPER-1823

zkTxnLogToolkit -dump should support printing transaction data as a string

Bug Closed Trivial Fixed maoling Raúl Gutiérrez Segalés Raúl Gutiérrez Segalés 27/Nov/13 20:44   20/May/19 13:50 12/Sep/18 07:33   3.6.0, 3.5.5 server   2 9 0 4200   Some times it's handy to have LogFormatter show you the content of the transactions (i.e.: if you are storing text). 100% 100% 4200 0 pull-request-available 360957 No Perforce job exists for this issue. 1 361256
1 year, 27 weeks, 1 day ago 0|i1q813:
ZooKeeper ZOOKEEPER-1821

very ugly warning when compiling load_gen.c

Bug Closed Major Fixed Germán Blanco Germán Blanco Germán Blanco 23/Nov/13 01:43   13/Mar/14 14:16 27/Nov/13 18:23 3.4.6, 3.5.0 3.4.6, 3.5.0 c client   0 5   macos this is the compiler output:
3.4/src/c/src/load_gen.c:110:1: warning: control may reach end of non-void function [-Wreturn-type]
[exec] }
[exec] ^
3.4/src/c/src/load_gen.c:135:1: warning: control may reach end of non-void function [-Wreturn-type]
[exec] }
[exec] ^
3.4/src/c/src/load_gen.c:163:1: warning: control may reach end of non-void function [-Wreturn-type]
[exec] }
[exec] ^
3.4/src/c/src/load_gen.c:180:1: warning: control may reach end of non-void function [-Wreturn-type]
[exec] }
[exec] ^
i think that the code is missing a "return ZOK" in the end of these functions.
360129 No Perforce job exists for this issue. 1 360428
6 years, 2 weeks ago 0|i1q2xz:
ZooKeeper ZOOKEEPER-1820

The artifact org.apache.zookeeper:zookeeper:pom:3.4.5 has no license specified.

Improvement Open Minor Unresolved Unassigned mhd wrk mhd wrk 21/Nov/13 19:14   21/Nov/13 19:14           1 1   We are using maven licensing pluging (org.linuxstuff.maven:licensing-maven-plugin) to check/enforce licensing requirements for our dependencies and noticed that zookeper doesn't include licensing terms in pom.xml which causes the plugin to report the above mentioned warning (see the summary).

Is there any plan to add the license to pom.xml as well?
359912 No Perforce job exists for this issue. 0 360211
6 years, 18 weeks ago 0|i1q1lr:
ZooKeeper ZOOKEEPER-1819

DeserializationPerfTest calls method with wrong arguments

Bug Resolved Minor Fixed Daniel Knightly Daniel Knightly Daniel Knightly 21/Nov/13 18:56   26/Apr/14 07:04 25/Apr/14 15:37 3.5.0 3.5.0 tests   0 4 3600 3600 0% The DeserializationPerfTest calls
SerializationPerfTest.createNodes to create serialized nodes to deserialize.

However 2 of the arguments, childcount and parentCVersion are switched in the call to the above method. This results in all tests unintentionally testing the same scenario.
0% 0% 3600 3600 359908 No Perforce job exists for this issue. 1 360207
5 years, 47 weeks, 5 days ago Fix DeserializationPerfTest which was not testing the desired scenarios. 0|i1q1kv:
ZooKeeper ZOOKEEPER-1818

Fix don't care for trunk

Bug Closed Blocker Fixed Fangmin Lv Flavio Paiva Junqueira Flavio Paiva Junqueira 16/Nov/13 06:53   20/May/19 13:50 17/Dec/18 09:20 3.5.1 3.6.0, 3.5.5     0 7 0 17400   See umbrella jira. 100% 100% 17400 0 pull-request-available 358924 No Perforce job exists for this issue. 1 359214
1 year, 13 weeks, 3 days ago This is very much a copy of the patch for 3.4.7 with small adaptations. 0|i1pvg7:
ZooKeeper ZOOKEEPER-1817

ZOOKEEPER-1805 Fix don't care for b3.4

Sub-task Closed Blocker Fixed Flavio Paiva Junqueira Flavio Paiva Junqueira Flavio Paiva Junqueira 16/Nov/13 06:53   13/Mar/14 14:16 23/Nov/13 13:24   3.4.6     0 3   See umbrella jira. 358923 No Perforce job exists for this issue. 13 359213
6 years, 2 weeks ago Thanks for the reviews and all the help, guys. Committed revision 1544858. 0|i1pvfz:
ZooKeeper ZOOKEEPER-1816

ClientCnxn.close() should block until threads have died

Bug Patch Available Minor Unresolved wu wen Jared Winick Jared Winick 15/Nov/13 16:33   22/Jun/18 00:49   3.3.6, 3.4.5, 3.4.11   java client   1 4   In the testing of ACCUMULO-1379 and ACCUMULO-1858 it was seen that the non-blocking behavior of ClientCnxn.close(), and therefore ZooKeeper.close(), can cause a race condition when undeploying an application running in a Java container such as JBoss or Tomcat. As the close() method returns without joining on the sendThread and eventThread, those threads continue to execute/cleanup while the container is cleaning up the application's resources. If the container has unloaded classes by the time this code runs

{code}
ZooTrace.logTraceMessage(LOG, ZooTrace.getTextTraceLevel(), "SendThread exitedloop.");
{code}

A "java.lang.NoClassDefFoundError: org/apache/zookeeper/server/ZooTrace" can be seen.
358855 No Perforce job exists for this issue. 1 359145
3 years, 19 weeks, 2 days ago 0|i1pv0v:
ZooKeeper ZOOKEEPER-1815

Tolerate incorrectly set system hostname in tests

Improvement Closed Trivial Fixed Unassigned some one some one 15/Nov/13 16:14   02/Apr/19 06:40 18/Nov/13 21:12   3.5.0, 3.4.14 tests   0 4 0 1800   A bunch of tests will fail with UnknownHostException errors when the hostname is incorrectly set on the system that you are running tests on. 100% 100% 1800 0 pull-request-available 358848 No Perforce job exists for this issue. 3 359138
6 years, 18 weeks, 2 days ago 0|i1puzb:
ZooKeeper ZOOKEEPER-1814

Reduction of waiting time during Fast Leader Election

Bug Patch Available Major Unresolved Daniel Peon Daniel Peon Daniel Peon 15/Nov/13 05:37   05/Feb/20 07:12   3.4.5, 3.5.0 3.7.0, 3.5.8 leaderElection   0 7 86400 86400 0% FastLeader election takes long time because of the exponential backoff. Currently the time is 60 seconds.

It would be interesting to give the possibility to configure this parameter, like for example for a Server shutdown.

Otherwise, it sometimes takes so long and it has been detected a test failure when executing: org.apache.zookeeper.server.quorum.QuorumPeerMainTest.

This test case waits until 30 seconds and this is smaller than the 60 seconds where the leader election can be waiting for at the moment of shutting down.

Considering the failure during the test case, this issue was considered a possible bug.
0% 0% 86400 86400 358733 No Perforce job exists for this issue. 11 359023
1 year, 17 weeks ago 0|i1pu9r:
ZooKeeper ZOOKEEPER-1813

Zookeeper restart fails due to missing node from snapshot

Bug Resolved Major Duplicate Unassigned Vinayakumar B Vinayakumar B 15/Nov/13 01:40   06/Dec/13 16:33 15/Nov/13 04:15 3.4.5, 3.5.0 3.5.0     0 3   Due to following exception Zookeeper restart is failing

{noformat}java.io.IOException: Failed to process transaction type: 1 error: KeeperErrorCode = NoNode for /test/subdir2/subdir2/subdir
at org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:183)
at org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:222)
at org.apache.zookeeper.server.ZooKeeperServer.loadData(ZooKeeperServer.java:255)
at org.apache.zookeeper.server.ZooKeeperServer.startdata(ZooKeeperServer.java:380)
at org.apache.zookeeper.server.NIOServerCnxnFactory.startup(NIOServerCnxnFactory.java:748)
at org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:111)
at org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:90)
at org.apache.zookeeper.server.ZooKeeperServerMainTest$2.run(ZooKeeperServerMainTest.java:218)
Caused by: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /test/subdir2/subdir2/subdir
at org.apache.zookeeper.server.persistence.FileTxnSnapLog.processTransaction(FileTxnSnapLog.java:268)
at org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:181)
... 7 more{noformat}
358703 No Perforce job exists for this issue. 1 358993
6 years, 18 weeks, 6 days ago 0|i1pu33:
ZooKeeper ZOOKEEPER-1812

ZooInspector reconnection always fails if first connection fails

Bug Closed Minor Fixed Benjamin Jaton Benjamin Jaton Benjamin Jaton 10/Nov/13 15:28   13/Mar/14 14:17 12/Nov/13 01:30 3.4.5, 3.5.0 3.4.6, 3.5.0 contrib   0 4   Steps to reproduce:

- Connect to localhost:2181 when ZooKeeper server is down. After a few seconds, ZooInspector warns that the connection has failed
- start the ZooKeeper server
- Reconnect to localhost:2181, ZooInspector will still not be able to connect to the server.

The workaround is to relaunch ZooInspector.
zooinspector 357845 No Perforce job exists for this issue. 2 358135
6 years, 2 weeks ago Fixed ZooInspector reconnection
Reviewed
0|i1posn:
ZooKeeper ZOOKEEPER-1811

The ZooKeeperSaslClient service name principal is hardcoded to "zookeeper"

Bug Closed Major Fixed Harsh J Harsh J Harsh J 07/Nov/13 00:37   23/Jul/15 14:54 10/Feb/14 16:33 3.4.5 3.4.6, 3.5.0 java client   0 6   The ClientCnxn class in ZK instantiates the ZooKeeperSaslClient with a hardcoded service name of "zookeeper". This causes all apps to fail in accessing ZK in a secure environment where the administrator has changed the principal name ZooKeeper runs as.

The service name should be configurable.
357308 No Perforce job exists for this issue. 1 357598
6 years, 2 weeks ago Adds a new system property "zookeeper.sasl.client.username" that can be used to configure the ZK Kerberos (SASL) client user principal name to something other than "zookeeper" (default) for any environments that use non-standard naming for its principals.
Reviewed
0|i1plhj:
ZooKeeper ZOOKEEPER-1810

Add version to FLE notifications for trunk

Bug Resolved Major Fixed Germán Blanco Flavio Paiva Junqueira Flavio Paiva Junqueira 06/Nov/13 11:03   15/Nov/18 12:53 07/Jul/14 23:26 3.5.0 3.5.0 leaderElection   0 8   The same as ZOOKEEPER-1808 but for trunk. 357171 No Perforce job exists for this issue. 8 357461
5 years, 37 weeks, 2 days ago 0|i1pkn3:
ZooKeeper ZOOKEEPER-1809

ephemeral node not deleted (or recreated?) after session expiry

Bug Resolved Minor Not A Problem Unassigned Shaun Senecal Shaun Senecal 06/Nov/13 00:06   06/Dec/13 16:32 06/Nov/13 22:54 3.4.5, 3.5.0       0 4   We have been running into a situation where we attempt to recreate our ephemeral nodes after a session expiry, only to find that the node already exists. Admittedly, this is only happening when we are aggresively killing and recreating sessions in a tight loop, but I thought it might point to a larger issue which may need to be addressed.

Attached is a small app which demonstrates the issue, and a log file (client and server in the same log) which shows the issue as it occured. Reproducing the bug is a tedious process of rerunning the test over and over again, but I have typically been able to reproduce it within 15mins of trying. The test app is using Curator, however, I think the issue is occuring at the ZK level since the logs clearly indicate the ephermal node is deleted after the session expiry.

Interesting bits from the log
{noformat}
...
2013/11/06 13:46:03,065 INFO [ConnectionStateManager-0] Recreating node: /test
...
2013/11/06 13:46:03,070 DEBUG [SyncThread:0] Processing request:: sessionid:0x1422bbb36d10002 type:create cxid:0x2 zxid:0x8 txntype:1 reqpath:n/a
...
2013/11/06 13:46:03,071 DEBUG [main] Closing client for session: 0x1422bbb36d10002
2013/11/06 13:46:03,075 INFO [ProcessThread(sid:0 cport:-1):] Processed session termination for sessionid: 0x1422bbb36d10002
2013/11/06 13:46:03,079 DEBUG [SyncThread:0] Processing request:: sessionid:0x1422bbb36d10002 type:closeSession cxid:0x1 zxid:0x9 txntype:-11 reqpath:n/a
2013/11/06 13:46:03,080 DEBUG [SyncThread:0] Deleting ephemeral node /test for session 0x1422bbb36d10002
2013/11/06 13:46:03,080 DEBUG [SyncThread:0] sessionid:0x1422bbb36d10002 type:closeSession cxid:0x1 zxid:0x9 txntype:-11 reqpath:n/a
...
2013/11/06 13:46:04,459 INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:43462] Client attempting to renew session 0x1422bbb36d10002 at /127.0.0.1:59559
2013/11/06 13:46:04,459 INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:43462] Invalid session 0x1422bbb36d10002 for client /127.0.0.1:59559, probably expired
2013/11/06 13:46:04,460 INFO [main-SendThread(localhost:43462)] Unable to reconnect to ZooKeeper service, session 0x1422bbb36d10002 has expired, closing socket connection
2013/11/06 13:46:04,460 DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:43462] Dropping request: No session with sessionid 0x1422bbb36d10002 exists, probably expired and removed
...
2013/11/06 13:46:04,463 INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:43462] Client attempting to establish new session at /127.0.0.1:59560
2013/11/06 13:46:04,466 DEBUG [SyncThread:0] Processing request:: sessionid:0x1422bbb36d10003 type:createSession cxid:0x0 zxid:0xa txntype:-10 reqpath:n/a
2013/11/06 13:46:04,466 DEBUG [SyncThread:0] sessionid:0x1422bbb36d10003 type:createSession cxid:0x0 zxid:0xa txntype:-10 reqpath:n/a
2013/11/06 13:46:04,467 INFO [SyncThread:0] Established session 0x1422bbb36d10003 with negotiated timeout 30000 for client /127.0.0.1:59560
...
2013/11/06 13:46:04,473 INFO [ConnectionStateManager-0] Recreating node: /test
2013/11/06 13:46:04,474 DEBUG [SyncThread:0] Processing request:: sessionid:0x1422bbb36d10003 type:exists cxid:0x2 zxid:0xfffffffffffffffe txntype:unknown reqpath:/___CURATOR_KILL_SESSION___15970538640754
2013/11/06 13:46:04,474 DEBUG [SyncThread:0] sessionid:0x1422bbb36d10003 type:exists cxid:0x2 zxid:0xfffffffffffffffe txntype:unknown reqpath:/___CURATOR_KILL_SESSION___15970538640754
2013/11/06 13:46:04,475 INFO [ProcessThread(sid:0 cport:-1):] Got user-level KeeperException when processing sessionid:0x1422bbb36d10003 type:create cxid:0x3 zxid:0xc txntype:-1 reqpath:n/a Error Path:/test Error:KeeperErrorCode = NodeExists for /test
2013/11/06 13:46:04,475 DEBUG [main-SendThread(localhost:43462)] Reading reply sessionid:0x1422bbb36d10003, packet:: clientPath:null serverPath:null finished:false header:: 2,3 replyHeader:: 2,11,-101 request:: '/___CURATOR_KILL_SESSION___15970538640754,T response::
2013/11/06 13:46:04,476 INFO [main] Initiating client connection, connectString=127.0.0.1:43462 sessionTimeout=10000 watcher=com.netflix.curator.test.KillSession$2@4067d00a sessionId=1422bbb36d10003 sessionPasswd=<hidden>
...
2013/11/06 13:46:04,479 ERROR [ConnectionStateManager-0] Failed to recreate ephemeral node
org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists for /test
at org.apache.zookeeper.KeeperException.create(KeeperException.java:119)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
at com.netflix.curator.framework.imps.CreateBuilderImpl$10.call(CreateBuilderImpl.java:625)
at com.netflix.curator.framework.imps.CreateBuilderImpl$10.call(CreateBuilderImpl.java:609)
at com.netflix.curator.RetryLoop.callWithRetry(RetryLoop.java:106)
at com.netflix.curator.framework.imps.CreateBuilderImpl.pathInForeground(CreateBuilderImpl.java:605)
at com.netflix.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:428)
at com.netflix.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:408)
at com.netflix.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:41)
at com.rakuten.sandbox.sessionexpiry.nodeexists.SessionExpiryTest$2.stateChanged(SessionExpiryTest.java:72)
at com.netflix.curator.framework.state.ConnectionStateManager$2.apply(ConnectionStateManager.java:184)
at com.netflix.curator.framework.state.ConnectionStateManager$2.apply(ConnectionStateManager.java:180)
at com.netflix.curator.framework.listen.ListenerContainer$1.run(ListenerContainer.java:92)
at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:262)
at com.netflix.curator.framework.listen.ListenerContainer.forEach(ListenerContainer.java:83)
at com.netflix.curator.framework.state.ConnectionStateManager.processEvents(ConnectionStateManager.java:177)
at com.netflix.curator.framework.state.ConnectionStateManager.access$000(ConnectionStateManager.java:40)
at com.netflix.curator.framework.state.ConnectionStateManager$1.call(ConnectionStateManager.java:104)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
...
{noformat}
357097 No Perforce job exists for this issue. 3 357387
6 years, 19 weeks, 2 days ago 0|i1pk6n:
ZooKeeper ZOOKEEPER-1808

ZOOKEEPER-1805 Add version to FLE notifications for 3.4 branch

Sub-task Closed Blocker Fixed Flavio Paiva Junqueira Flavio Paiva Junqueira Flavio Paiva Junqueira 05/Nov/13 12:03   13/Mar/14 14:16 15/Nov/13 13:27   3.4.6     0 5   Add version to notification messages so that we can differentiate messages during rolling upgrades. This task is for the 3.4 branch only. 356983 No Perforce job exists for this issue. 8 357273
6 years, 2 weeks ago 0|i1pjhr:
ZooKeeper ZOOKEEPER-1807

Observers spam each other creating connections to the election addr

Bug Resolved Blocker Fixed Alexander Shraer Raúl Gutiérrez Segalés Raúl Gutiérrez Segalés 01/Nov/13 19:32   10/May/18 16:01 10/May/18 16:01   3.5.4, 3.6.0     0 11   Hey [~shralex],

I noticed today that my Observers are spamming each other trying to open connections to the election port. I've got tons of these:

{noformat}
2013-11-01 22:19:45,819 - DEBUG [WorkerSender[myid=13]] - There is a connection already for server 9
2013-11-01 22:19:45,819 - DEBUG [WorkerSender[myid=13]] - There is a connection already for server 10
2013-11-01 22:19:45,819 - DEBUG [WorkerSender[myid=13]] - There is a connection already for server 6
2013-11-01 22:19:45,819 - DEBUG [WorkerSender[myid=13]] - There is a connection already for server 12
2013-11-01 22:19:45,819 - DEBUG [WorkerSender[myid=13]] - There is a connection already for server 14
{noformat}

and so and so on ad nauseam.

Now, looking around I found this inside FastLeaderElection.java from when you committed ZOOKEEPER-107:

{noformat}
private void sendNotifications() {
- for (QuorumServer server : self.getVotingView().values()) {
- long sid = server.id;
-
+ for (long sid : self.getAllKnownServerIds()) {
+ QuorumVerifier qv = self.getQuorumVerifier();
{noformat}

Is that really desired? I suspect that is what's causing Observers to try to connect to each other (as opposed as just connecting to participants). I'll give it a try now and let you know. (Also, we use observer ids that are > 0, and I saw some parts of the code that might not deal with that assumption - so it could be that too..).
356550 No Perforce job exists for this issue. 9 356838
1 year, 45 weeks ago 0|i1pgtb:
ZooKeeper ZOOKEEPER-1806

testCurrentServersAreObserversInNextConfig failing frequently on trunk with non-jdk6

Bug Closed Major Duplicate Alexander Shraer Patrick D. Hunt Patrick D. Hunt 31/Oct/13 17:47   19/Dec/19 18:01 10/Aug/16 13:04 3.5.0 3.5.3 server   0 3   testCurrentServersAreObserversInNextConfig failing frequently on trunk with jdk7

I see a number of failures recently on this test. Is it a real issue or flakey? Perhaps due to re-ordering/cleanup with jdk7 as we've seen with some other tests? (I don't see this test failing with jdk6)
test 356338 No Perforce job exists for this issue. 0 356626
3 years, 32 weeks, 1 day ago 0|i1pfi7:
ZooKeeper ZOOKEEPER-1805

"Don't care" value in ZooKeeper election breaks rolling upgrades

Bug Closed Blocker Fixed Flavio Paiva Junqueira Flavio Paiva Junqueira Flavio Paiva Junqueira 29/Oct/13 23:18   13/Mar/14 14:31 13/Mar/14 14:25   3.4.6     1 9   ZOOKEEPER-1808, ZOOKEEPER-1817 This is an issue that has been originally reported in ZOOKEEPER-1732. 355971 No Perforce job exists for this issue. 11 356259
6 years, 2 weeks ago 0|i1pd93:
ZooKeeper ZOOKEEPER-1804

Stat the realtime tps of zookeepr server

Improvement Open Major Unresolved Leader Ni Leader Ni Leader Ni 29/Oct/13 04:56   14/Dec/19 06:07     3.7.0 server 08/Nov/13 0 6   At this time, we assessed whether zookeeper supports some business scenarios, always use the number of subscribers, or to assess the number of clients。

You konw, some times, many client connection with zookeeper, but do noting, and the onthers do complex business logic。

So,we must stat the realtime tps of zookeepr。


[-----------------Solution-------------------]

Solution1:

If you only want to know the real time transaction processed, you can use the patch "ZOOKEEPER-1804.patch".

Solution2:

If you also want to know how client use zookeeper, and the real time r/w ps of each zookeeper client, you can use the patch "ZOOKEEPER-1804-2.patch"
use java properties: -Dserver_process_stats=true to open the function.

Sample:
$>echo rwps|nc localhost 2181
RealTime R/W Statistics:
getChildren2:   0.5994005994005994
createSession:  1.6983016983016983
closeSession:   0.999000999000999
setData:     110.18981018981019
setWatches:   129.17082917082917
getChildren:    68.83116883116884
delete:      19.980019980019982
create:      22.27772227772228
exists:      1806.2937062937062
getDate:     729.5704295704296
355792 No Perforce job exists for this issue. 2 356080
1 year, 7 weeks, 1 day ago 0|i1pc5b:
ZooKeeper ZOOKEEPER-1803

Add description for pzxid in programmer's guide.

Bug Closed Major Fixed Mohammad Arshad Leader Ni Leader Ni 26/Oct/13 07:50   21/Jul/16 16:18 25/Sep/15 02:52   3.4.7, 3.5.2, 3.6.0 documentation 31/Oct/13 0 6   The Stat(org.apache.zookeeper.data.Stat) Structures has the filed pzxid, but no document about it in programmer's guide(http://zookeeper.apache.org/doc/r3.4.3/zookeeperProgrammers.html#sc_zkStatStructure) 355442 No Perforce job exists for this issue. 2 355730
4 years, 25 weeks, 6 days ago 0|i1p9zr:
ZooKeeper ZOOKEEPER-1802

ZOOKEEPER-3170 flaky test testResyncByTxnlogThenDiffAfterFollowerCrashes

Sub-task Resolved Major Cannot Reproduce Andor Molnar Patrick D. Hunt Patrick D. Hunt 25/Oct/13 12:48   25/Oct/18 11:29 25/Oct/18 11:29 3.5.0   tests   0 2   This test fails intermittently on trunk:
https://builds.apache.org/view/S-Z/view/ZooKeeper/job/ZooKeeper-trunk-jdk7/691/testReport/junit/org.apache.zookeeper.test/FollowerResyncConcurrencyTest/testResyncByTxnlogThenDiffAfterFollowerCrashes/
flaky 355344 No Perforce job exists for this issue. 0 355632
1 year, 21 weeks ago 0|i1p9dz:
ZooKeeper ZOOKEEPER-1801

TestReconfig failure

Bug Open Major Unresolved Marshall McMullen Flavio Paiva Junqueira Flavio Paiva Junqueira 25/Oct/13 08:16   05/Feb/20 07:16     3.7.0, 3.5.8 quorum   0 3   This is the message:

{noformat}
/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk/trunk/src/c/tests/TestReconfig.cc:183: Assertion: equality assertion failed [Expected: 1, Actual : 0]
{noformat}

https://builds.apache.org/job/ZooKeeper-trunk/2100/
355300 No Perforce job exists for this issue. 0 355588
1 year, 17 weeks, 1 day ago 0|i1p947:
ZooKeeper ZOOKEEPER-1800

jenkins failure in testGetProposalFromTxn

Bug Open Major Unresolved Thawan Kooburat Patrick D. Hunt Patrick D. Hunt 24/Oct/13 13:04   05/Feb/20 07:17   3.5.0 3.7.0, 3.5.8 tests   0 1   https://builds.apache.org/view/S-Z/view/ZooKeeper/job/ZooKeeper-trunk-jdk7/691/testReport/junit/org.apache.zookeeper.test/GetProposalFromTxnTest/testGetProposalFromTxn/

test was introduced in ZOOKEEPER-1413, seems to have failed twice so far this month.
355096 No Perforce job exists for this issue. 0 355384
6 years, 21 weeks, 2 days ago 0|i1p7uv:
ZooKeeper ZOOKEEPER-1799

SaslAuthFailDesignatedClientTest.testAuth fails frequently on SUSE

Bug Closed Minor Fixed Jeffrey Zhong Jeffrey Zhong Jeffrey Zhong 22/Oct/13 14:20   13/Mar/14 14:17 22/Oct/13 19:17 3.4.5 3.4.6, 3.5.0 tests   0 4   org.apache.zookeeper.test.SaslAuthFailDesignatedClientTest.testAuth often fails on SUSE with the following error stack trace:
{code}
junit.framework.AssertionFailedError: expected [0x141ccb60d870000] expected:<1> but was:<0>
at org.apache.zookeeper.test.JMXEnv.ensureAll(JMXEnv.java:115)
at org.apache.zookeeper.test.ClientBase.createClient(ClientBase.java:200)
at org.apache.zookeeper.test.ClientBase.createClient(ClientBase.java:174)
at org.apache.zookeeper.test.ClientBase.createClient(ClientBase.java:159)
at org.apache.zookeeper.test.ClientBase.createClient(ClientBase.java:152)
at org.apache.zookeeper.test.SaslAuthFailDesignatedClientTest.testAuth(SaslAuthFailDesignatedClientTest.java:87)
at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:52)
{code}

The reason is that this is a negative test. After authentication fails, the client connection is closed at the server side so does the session right before test case calls JMXEnv.ensureAll to verify the session. Below are the log events show the sequence and you can see the session was already closed before client JMXEnv.ensureAll.

{code}
2013-10-18 10:56:25,320 [myid:] - INFO [SyncThread:0:ZooKeeperServer@595] - Established session 0x141ccb60d870000 with negotiated timeout 30000 for client /127.0.0.1:58272
2013-10-18 10:56:25,327 [myid:] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11221:ZooKeeperServer@940] - Client failed to SASL authenticate: javax.security.sasl.SaslException: DIGEST-MD5: digest response format violation. Mismatched response.
2013-10-18 10:56:25,327 [myid:] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11221:ZooKeeperServer@946] - Closing client connection due to SASL authentication failure.
2013-10-18 10:56:25,329 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11221:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:58272 which had sessionid 0x141ccb60d870000
....
2013-10-18 10:56:25,330 [myid:] - INFO [main-SendThread(localhost:11221):ClientCnxn$SendThread@1089] - Unable to read additional data from server sessionid 0x141ccb60d870000, likely server has closed socket, closing socket connection and attempting reconnect
2013-10-18 10:56:25,332 [myid:] - INFO [main:JMXEnv@105] - expect:0x141ccb60d870000
{code}
354710 No Perforce job exists for this issue. 1 354999
6 years, 2 weeks ago
Reviewed
0|i1p5hb:
ZooKeeper ZOOKEEPER-1798

Fix race condition in testNormalObserverRun

Bug Closed Blocker Fixed Thawan Kooburat Flavio Paiva Junqueira Flavio Paiva Junqueira 20/Oct/13 04:52   26/Aug/16 17:16 13/Nov/13 23:47   3.4.6, 3.5.0     0 4   This is the output messges:

<noformat>
Testcase: testNormalObserverRun took 4.221 sec
FAILED
expected:<data[2]> but was:<data[1]>
junit.framework.AssertionFailedError: expected:<data[2]> but was:<data[1]>
at org.apache.zookeeper.server.quorum.Zab1_0Test$8.converseWithObserver(Zab1_0Test.java:1118)
at org.apache.zookeeper.server.quorum.Zab1_0Test.testObserverConversation(Zab1_0Test.java:546)
at org.apache.zookeeper.server.quorum.Zab1_0Test.testNormalObserverRun(Zab1_0Test.java:994)
<noformat>
354303 No Perforce job exists for this issue. 7 354593
6 years, 2 weeks ago 0|i1p2zb:
ZooKeeper ZOOKEEPER-1797

PurgeTxnLog may delete data logs during roll

Bug Resolved Blocker Fixed Rakesh Radhakrishnan Derek Dagit Derek Dagit 18/Oct/13 14:57   10/Sep/16 20:06 17/May/14 21:05 3.4.5 3.4.7, 3.5.0 server   0 8   org.apache.zookeeper.server.PurgeTxnLog deletes old data logs and snapshots, keeping the newest N snapshots and any data logs that have been written since the snapshot.

It does this by listing the available snapshots & logs and creates a blacklist of snapshots and logs that should not be deleted. Then, it searches for and deletes all logs and snapshots that are not in this list.

It appears that if logs are rolling or a new snapshot is created during this process, then these newer files will be unintentionally deleted.
354189 No Perforce job exists for this issue. 4 354481
4 years, 49 weeks, 2 days ago 0|i1p2an:
ZooKeeper ZOOKEEPER-1796

Move common code from {Follower, Observer}ZooKeeperServer into LearnerZooKeeperServer

Improvement Resolved Trivial Fixed Raúl Gutiérrez Segalés Raúl Gutiérrez Segalés Raúl Gutiérrez Segalés 14/Oct/13 18:31   02/Mar/16 20:32 21/Mar/14 01:37   3.5.0     0 3   Since ZOOKEEPER-1552 we are enabling syncProcessor in Observers, so we should have a proper shutdown() method there. Since FollowerZooKeeperServer already has one, which does the same thing that we need, move that to LearnerZooKeeperServer along with some related instance variables. 353404 No Perforce job exists for this issue. 1 353696
6 years, 6 days ago 0|i1oxhz:
ZooKeeper ZOOKEEPER-1795

unable to build c client on ubuntu

Bug Resolved Blocker Fixed Raúl Gutiérrez Segalés Patrick D. Hunt Patrick D. Hunt 14/Oct/13 15:38   06/Dec/13 16:30 14/Oct/13 19:57 3.5.0 3.5.0 c client   0 4   Seems there is an issue for Ubuntu (I'm on 13.04), however I'm only seeing it on trunk and not branch 34

{noformat}
make check
make zktest-st zktest-mt
make[1]: Entering directory `/home/phunt/dev/svn/svn-zookeeper/src/c'
g++ -DHAVE_CONFIG_H -I. -I./include -I./tests -I./generated -DUSE_STATIC_LIB -DZKSERVER_CMD="\"./tests/zkServer.sh\"" -DZOO_IPV6_ENABLED -g -O2 -MT zktest_st-TestReconfigServer.o -MD -MP -MF .deps/zktest_st-TestReconfigServer.Tpo -c -o zktest_st-TestReconfigServer.o `test -f 'tests/TestReconfigServer.cc' || echo './'`tests/TestReconfigServer.cc
tests/TestReconfigServer.cc: In member function 'bool TestReconfigServer::waitForConnected(zhandle_t*, uint32_t)':
tests/TestReconfigServer.cc:128:16: error: 'sleep' was not declared in this scope
make[1]: *** [zktest_st-TestReconfigServer.o] Error 1
make[1]: Leaving directory `/home/phunt/dev/svn/svn-zookeeper/src/c'
make: *** [check-am] Error 2
{noformat}

I have

{noformat}
g++ --version
g++ (Ubuntu/Linaro 4.7.3-1ubuntu1) 4.7.3
Copyright (C) 2012 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
{noformat}
353379 No Perforce job exists for this issue. 1 353671
6 years, 23 weeks, 2 days ago
Reviewed
0|i1oxcf:
ZooKeeper ZOOKEEPER-1794

ZOOKEEPER-1777 Add hash check to transaction history in quorum servers

Sub-task Open Major Unresolved Germán Blanco Germán Blanco Germán Blanco 14/Oct/13 02:15   14/Dec/19 06:09   3.5.0 3.7.0 quorum   1 6 1209600 1209600 0% The goal of this task is to add a hash number to each transaction in the transaction history. This hash number will depend on the entire transaction history. This hash number will be the same in all members of the quorum, since it shall have the same result if the members have the same transaction history. That means that there will be no need to send any new information between members of the quorum, during the broadcast phase. The hash number will be checked by the leader when learners try to connect, and it shall also be sent together with the snapshot during synchronisation. If the hash number does not match, the synchronisation shall be done with a snapshot in order to overwrite the conflicts in the transaction history. 0% 0% 1209600 1209600 353270 No Perforce job exists for this issue. 3 353563
5 years, 49 weeks, 3 days ago The snapshot file name will be updated by adding also the hash. That means a name like "snapshot.100001.123533" where 100001 is the zxid and 123533 is the hash. The server will continue to be able to read files without the hash in the name. 0|i1owof:
ZooKeeper ZOOKEEPER-1793

Zab1_0Test.testNormalObserverRun() is flaky

Bug Resolved Major Duplicate Unassigned Alexander Shraer Alexander Shraer 11/Oct/13 00:43   06/Dec/13 16:30 24/Oct/13 14:54     quorum, server, tests   0 4   not sure if this is due to a known issue or not.

// check and make sure the change is persisted
zkDb2 = new ZKDatabase(new FileTxnSnapLog(logDir, snapDir));
lastZxid = zkDb2.loadDataBase();
Assert.assertEquals("data2", new String(zkDb2.getData("/foo", stat, null)));

this assert periodically (once every 3 runs of the test or so) fails saying that getData returns data1 and not data2.
352988 No Perforce job exists for this issue. 0 353275
6 years, 22 weeks ago 0|i1ouwf:
ZooKeeper ZOOKEEPER-1792

Observers don't need to keep an in-memory copy of last commited proposals

Improvement Open Minor Unresolved Unassigned Raúl Gutiérrez Segalés Raúl Gutiérrez Segalés 10/Oct/13 21:18   29/Jun/15 01:55           0 2   In FinalRequestProcessor.java#processRequest we have:

{noformat}
if (request.isQuorum()) {
zks.getZKDatabase().addCommittedProposal(request);
}
{noformat}

but this is only useful to the leader since committed proposals are only used from LearnerHandler to sync up followers. I presume followers do need it as they might become a leader at any point. But observers have no need for them, so we could probably special case this for them and optimize the path for them.
352973 No Perforce job exists for this issue. 0 353260
4 years, 38 weeks, 3 days ago 0|i1out3:
ZooKeeper ZOOKEEPER-1791

ZooKeeper package includes unnecessary jars that are part of the package.

Bug Resolved Major Fixed Mahadev Konar Mahadev Konar Mahadev Konar 10/Oct/13 20:07   20/May/14 07:09 17/May/14 22:42 3.5.0 3.5.0 build   0 4   ZooKeeper package includes unnecessary jars that are part of the package.

Packages like fatjar and

{code}
maven-ant-tasks-2.1.3.jar
maven-artifact-2.2.1.jar
maven-artifact-manager-2.2.1.jar
maven-error-diagnostics-2.2.1.jar
maven-model-2.2.1.jar
maven-plugin-registry-2.2.1.jar
maven-profile-2.2.1.jar
maven-project-2.2.1.jar
maven-repository-metadata-2.2.1.jar
{code}

are part of the zookeeper package and rpm (via bigtop).
352961 No Perforce job exists for this issue. 1 353248
5 years, 44 weeks, 2 days ago 0|i1ouqf:
ZooKeeper ZOOKEEPER-1790

Deal with special ObserverId in QuorumCnxManager.receiveConnection

Bug Closed Major Not A Problem Alexander Shraer Alexander Shraer Alexander Shraer 10/Oct/13 19:59   13/Mar/14 14:16 13/Oct/13 01:50 3.4.6, 3.5.0 3.4.6, 3.5.0 server   0 4   QuorumCnxManager.receiveConnection assumes that a negative sid means that this is a 3.5.0 server, which has a different communication protocol. This doesn't account for the fact that ObserverId = -1 is a special id that may be used by observers and is also negative.

This requires a fix to trunk and a separate fix to 3.4 branch, where this function is different (see ZOOKEEPER-1633)
352959 No Perforce job exists for this issue. 0 353246
6 years, 2 weeks ago 0|i1oupz:
ZooKeeper ZOOKEEPER-1789

3.4.x observer causes NPE on 3.5.0 (trunk) participants

Bug Resolved Critical Fixed Alexander Shraer Raúl Gutiérrez Segalés Raúl Gutiérrez Segalés 10/Oct/13 19:10   24/Jul/14 07:22 23/Jul/14 13:36 3.5.0 3.5.0     0 6   (assigning to Alex because this was introduced by ZOOKEEPER-107, but will upload a patch as well.)

I have a 5 participants cluster running what will be 3.5.0 (i.e.: trunk as of today) and an observer running 3.4 (trunk from 3.4 branch). When the observer tries to establish a connection to the participants I get:

{noformat}
Thread Thread[10.40.78.121:3888,5,main] died java.lang.NullPointerException at org.apache.zookeeper.server.quorum.QuorumCnxManager.receiveConnection(QuorumCnxManager.java:240)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$Listener.run(QuorumCnxManager.java:552)
{noformat}

Looking at QuorumCnxManager.java:240:

{noformat}
if (protocolVersion >= 0) { // this is a server id and not a protocol version
sid = protocolVersion;
electionAddr = self.getVotingView().get(sid).electionAddr;
} else {
{noformat}

and self.getVotingView().get(sid) will be null for Observers. So this block should cover that case.
352951 No Perforce job exists for this issue. 2 353238
5 years, 35 weeks ago
Reviewed
0|i1ouo7:
ZooKeeper ZOOKEEPER-1788

Support clientID field on connection requests

Improvement Open Minor Unresolved Unassigned Raúl Gutiérrez Segalés Raúl Gutiérrez Segalés 09/Oct/13 15:43   09/Oct/13 15:43           0 1   I suspect it's very common for deployments to have a wide variety of client libraries (different versions/languages) connecting to a given cluster.

It would be handy to have a way to identify clients via a clientID (akin to HTTP's User-Agent header). This could be implemented in ZooKeeperServer#processConnectRequest [1] and be fully backwards compatible.

The clientID could then be kept with the corresponding ServerCnxn instance and be used for better logging (or stats expose through 4-letter commands).

The corresponding client side change would be to expose API to set the clientID on each connection handler (and by default it could be something like "zk java $version", "zk c $version", etc).

Thoughts?

[1] https://github.com/apache/zookeeper/blob/trunk/src/java/main/org/apache/zookeeper/server/ZooKeeperServer.java#L797
352722 No Perforce job exists for this issue. 0 353009
6 years, 24 weeks, 1 day ago 0|i1ot9r:
ZooKeeper ZOOKEEPER-1787

Add support for enabling local session in rolling upgrade

Bug Patch Available Minor Unresolved Raúl Gutiérrez Segalés Thawan Kooburat Thawan Kooburat 09/Oct/13 14:31   23/Aug/14 11:43   3.5.0   server   0 5   Currently, local session need to be enable by stopping the entire ensemble. If a rolling upgrade is used, all write request from a local session will fail with session move until the local session is enabled on the leader. 352703 No Perforce job exists for this issue. 1 352990
5 years, 30 weeks, 5 days ago 0|i1ot5r:
ZooKeeper ZOOKEEPER-1786

ZooKeeper data model documentation is incorrect

Bug Closed Minor Fixed Niraj Tolia Niraj Tolia Niraj Tolia 09/Oct/13 04:10   13/Mar/14 14:16 15/Nov/13 13:07 3.4.6 3.4.6, 3.5.0 documentation   0 4   When I look at https://zookeeper.apache.org/doc/trunk/zookeeperProgrammers.html#ch_zkDataModel, I see two things that seem wrong in terms of restricted characters:

* \uXFFFE - \uXFFFF (where X is a digit 1 - E)
* \uF0000 - \uFFFFF

These definitions are invalid characters in Java and aren't reflected in PathUtils either (or PathUtilsTest). In fact the code in PathUtils states:
{code:borderStyle=solid}
} else if (c > '\u0000' && c <= '\u001f'
|| c >= '\u007f' && c <= '\u009F'
|| c >= '\ud800' && c <= '\uf8ff'
|| c >= '\ufff0' && c <= '\uffff') {
reason = "invalid charater @" + i;
break;
}
{code}

Unless I am missing something, this simple patch should fix the documentation problem:
{code}
Index: src/docs/src/documentation/content/xdocs/zookeeperProgrammers.xml
===================================================================
--- src/docs/src/documentation/content/xdocs/zookeeperProgrammers.xml (revision 1530514)
+++ src/docs/src/documentation/content/xdocs/zookeeperProgrammers.xml (working copy)
@@ -139,8 +139,7 @@

<listitem>
<para>The following characters are not allowed: \ud800 - uF8FF,
- \uFFF0 - uFFFF, \uXFFFE - \uXFFFF (where X is a digit 1 - E), \uF0000 -
- \uFFFFF.</para>
+ \uFFF0 - uFFFF.</para>
</listitem>

<listitem>
{code}
352602 No Perforce job exists for this issue. 1 352889
6 years, 2 weeks ago 0|i1osjb:
ZooKeeper ZOOKEEPER-1785

Small fix in zkServer.sh to support new configuration format

Bug Resolved Minor Not A Problem Alexander Shraer Alexander Shraer Alexander Shraer 08/Oct/13 20:22   29/Mar/14 18:48 29/Mar/14 18:48 3.5.0 3.5.0 scripts   0 2   The problem can be reproduced by running a server with the following type of config file:

dataDir=/Users/shralex/zookeeper-test/zookeeper1
syncLimit=2
initLimit=5
tickTime=2000
server.1=localhost:2721:2731:participant;2791
server.2=localhost:2722:2732:participant;2792

and then trying to do "zkServer.sh status"

Here I specified the servers using the new config format but still used the static config file and didn't include the "clientPort" key.

zkServer.sh already supports the new configuration format, but expects server spec to appear in the dynamic config file if it uses the new format.
So in the example above it will not find the client port.

The current logic for executing something like 'zkServer.sh status' is:

1. Look for clientPort keyword in the static config file
2. Look for the client port in the server spec in the dynamic config file

The attached patch adds an intermediate step:
1'. Look for the client port in the server spec in the static config file

352553 No Perforce job exists for this issue. 1 352840
5 years, 51 weeks, 5 days ago 0|i1os8f:
ZooKeeper ZOOKEEPER-1784

Logic to process INFORMANDACTIVATE packets in syncWithLeader seems bogus

Bug Resolved Major Fixed Raúl Gutiérrez Segalés Raúl Gutiérrez Segalés Raúl Gutiérrez Segalés 08/Oct/13 15:18   07/Apr/15 17:15 24/Jan/15 19:23 3.5.0 3.5.1     0 6   If you look at Learner#syncWithLeader:

{noformat}
while (self.isRunning()) {
readPacket(qp);
switch(qp.getType()) {
.......

case Leader.INFORM:
case Leader.INFORMANDACTIVATE:
PacketInFlight packet = new PacketInFlight();
packet.hdr = new TxnHeader();

if (qp.getType() == Leader.COMMITANDACTIVATE) {
{noformat}

I guess "qp.getType() == Leader.COMMITANDACTIVATE" is a typo that should read "qp.getType() == Leader.INFORMANDACTIVATE".

Assigning to Alexander for now since this is part of ZOOKEEPER-107.
352492 No Perforce job exists for this issue. 2 352779
4 years, 50 weeks, 2 days ago Committed to trunk. Thanks Raul, and sorry that it took so long to commit. 0|i1oruv:
ZooKeeper ZOOKEEPER-1783

Distinguish initial configuration from first established configuration

Bug Resolved Major Fixed Alexander Shraer Alexander Shraer Alexander Shraer 07/Oct/13 14:01   29/Mar/14 18:19 07/Nov/13 12:07 3.5.0 3.5.0 quorum, server   0 4   We need a way to distinguish an initial config of a server and an initial config of a running ensemble (before any reconfigs happen). Currently both have version 0.

The version of a config increases with each reconfiguration, so the problem is just with the initial config.
352283 No Perforce job exists for this issue. 10 352571
6 years, 20 weeks ago 0|i1oqkn:
ZooKeeper ZOOKEEPER-1782

zookeeper.superUser is not as super as superDigest

Bug Resolved Major Fixed Robert Joseph Evans Robert Joseph Evans Robert Joseph Evans 04/Oct/13 14:24   23/Jun/17 13:45 23/Jun/17 12:40 3.4.5 3.5.4, 3.6.0     1 7   The zookeeper.superUser system property does not fully grant super user privileges, like zookeeper.DigestAuthenticationProvider.superDigest does.

zookeeper.superUser only has as many privileges as the sasl ACLs on the znode being accessed. This means that if a znode only has digest ACLs zookeeper.superUser is ignored. Or if a znode has a single sasl ACL that only has read privileges zookeeper.superUser only has read privileges.

The reason for this is that SASLAuthenticationProvider implements the superUser check in the matches method, instead of having the super user include a new Id("super","") as Digest does.
352012 No Perforce job exists for this issue. 2 352300
2 years, 38 weeks, 6 days ago 0|i1oown:
ZooKeeper ZOOKEEPER-1781

ZooKeeper Server fails if snapCount is set to 1

Bug Closed Minor Fixed Takashi Ohnishi Takashi Ohnishi Takashi Ohnishi 03/Oct/13 21:19   13/Mar/14 14:17 07/Oct/13 19:38 3.4.5 3.4.6, 3.5.0 quorum   0 5   If snapCount is set to 1, ZooKeeper Server can start but it fails with the below error:

2013-10-02 18:09:07,600 [myid:1] - ERROR [SyncThread:1:SyncRequestProcessor@151] - Severe unrecoverable error, exiting
java.lang.IllegalArgumentException: n must be positive
at java.util.Random.nextInt(Random.java:300)
at org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:93)

In source code, it maybe be supposed that snapCount must be 2 or more:
{code:title=org.apache.zookeeper.server.SyncRequestProcessor.java|borderStyle=solid}
91 // we do this in an attempt to ensure that not all ofthe servers
92 // in the ensemble take a snapshot at the same time
93 int randRoll = r.nextInt(snapCount/2);
{code}

I think this supposition is not bad because snapCount = 1 is not realistic setting...
But, it may be better to mention this restriction in documentation or add a validation in the source code.
351893 No Perforce job exists for this issue. 2 352181
6 years, 2 weeks ago
Reviewed
0|i1oo6n:
ZooKeeper ZOOKEEPER-1780

add a FAQ for why myid file is required

Improvement Open Minor Unresolved Unassigned Patrick D. Hunt Patrick D. Hunt 02/Oct/13 14:56   02/Oct/13 14:56           0 1   This comes up every so often, would be good for us to document on the FAQ.

https://cwiki.apache.org/confluence/display/ZOOKEEPER/FAQ

here's one such discussion: http://markmail.org/message/cvzz3tq3gievicqe
351629 No Perforce job exists for this issue. 0 351917
6 years, 25 weeks, 1 day ago 0|i1omk7:
ZooKeeper ZOOKEEPER-1779

ReconfigTest littering the source root with test files

Bug Resolved Critical Fixed Abhiraj Butala Patrick D. Hunt Patrick D. Hunt 02/Oct/13 14:42   06/Mar/14 06:09 05/Mar/14 16:55 3.5.0 3.5.0 tests   0 4   After running the ReconfigTest I saw a number of the following files in the source root (not in a subdir of the build directory as would be expected)

zoo_replicated1.dynamic
(saw files zoo_replicated{1-9})
351625 No Perforce job exists for this issue. 1 351913
6 years, 3 weeks ago 0|i1omjb:
ZooKeeper ZOOKEEPER-1778

Use static final Logger objects

Improvement Resolved Minor Fixed Rakesh Radhakrishnan Rakesh Radhakrishnan Rakesh Radhakrishnan 02/Oct/13 13:36   05/Oct/13 07:10 04/Oct/13 18:58   3.5.0     0 5   Logger is not declared as 'private static final' in few classes 351617 No Perforce job exists for this issue. 1 351906
6 years, 24 weeks, 5 days ago 0|i1omhr:
ZooKeeper ZOOKEEPER-1777

Missing ephemeral nodes in one of the members of the ensemble

Bug Open Major Unresolved Germán Blanco Germán Blanco Germán Blanco 02/Oct/13 06:09   14/Dec/19 06:09   3.4.5 3.7.0 quorum   0 7   ZOOKEEPER-1794 Linux, Java 1.7 In a 3-servers ensemble, one of the followers doesn't see part of the ephemeral nodes that are present in the leader and the other follower.
The 8 missing nodes in "the follower that is not ok" were created in the end of epoch 1, the ensemble is running in epoch 2.
0% 1209600 1209600 351470 No Perforce job exists for this issue. 6 351759
6 years, 22 weeks ago 0|i1oll3:
ZooKeeper ZOOKEEPER-1776

Ephemeral nodes not present in one of the members of the ensemble

Bug Closed Major Invalid Germán Blanco Germán Blanco Germán Blanco 02/Oct/13 06:07   13/Mar/14 14:17 02/Oct/13 06:51 3.4.5 3.4.6, 3.5.0 quorum   0 2   Linux, Java 1.7 In a 3-servers ensemble, one of the followers doesn't see part of the ephemeral nodes that are present in the leader and the other follower.
The 8 missing nodes in "the follower that is not ok" were created in the end of epoch 1, the ensemble is running in epoch 2.
351469 No Perforce job exists for this issue. 0 351758
6 years, 2 weeks ago 0|i1olkv:
ZooKeeper ZOOKEEPER-1775

Ephemeral nodes not present in one of the members of the ensemble

Bug Closed Major Invalid Germán Blanco Germán Blanco Germán Blanco 02/Oct/13 06:05   13/Mar/14 14:17 02/Oct/13 06:52 3.4.5 3.4.6, 3.5.0 quorum   0 2   Linux, Java 1.7 In a 3-servers ensemble, one of the followers doesn't see part of the ephemeral nodes that are present in the leader and the other follower.
The 8 missing nodes in "the follower that is not ok" were created in the end of epoch 1, the ensemble is running in epoch 2.
351467 No Perforce job exists for this issue. 0 351756
6 years, 2 weeks ago 0|i1olkf:
ZooKeeper ZOOKEEPER-1774

QuorumPeerMainTest fails consistently with "complains about host" assertion failure

Bug Closed Blocker Fixed Patrick D. Hunt Patrick D. Hunt Patrick D. Hunt 01/Oct/13 19:14   13/Mar/14 14:17 08/Oct/13 01:28 3.4.6 3.4.6, 3.5.0 quorum, tests   0 6   Ubuntu 13.04

Linux version 3.8.0-30-generic (buildd@akateko) (gcc version 4.7.3 (Ubuntu/Linaro 4.7.3-1ubuntu1) ) #44-Ubuntu SMP Thu Aug 22 20:54:42 UTC 2013

java -version
java version "1.6.0_45"
Java(TM) SE Runtime Environment (build 1.6.0_45-b06)
Java HotSpot(TM) Server VM (build 20.45-b01, mixed mode)
QuorumPeerMainTest fails consistently with "complains about host" assertion failure.


{noformat}
2013-10-01 16:09:17,962 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@54] - TEST METHOD FAILED testBadPeerAddressInQuorum
java.lang.AssertionError: complains about host
at org.junit.Assert.fail(Assert.java:91)
at org.junit.Assert.assertTrue(Assert.java:43)
at org.apache.zookeeper.server.quorum.QuorumPeerMainTest.testBadPeerAddressInQuorum(QuorumPeerMainTest.java:434)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:52)
at org.junit.rules.TestWatchman$1.evaluate(TestWatchman.java:48)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:76)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184)
at org.junit.runners.ParentRunner.run(ParentRunner.java:236)
at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:39)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:518)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1052)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:906)
2013-10-01 16:09:17,963 [myid:] - INFO [main:ZKTestCase$1@65] - FAILED testBadPeerAddressInQuorum
java.lang.AssertionError: complains about host
at org.junit.Assert.fail(Assert.java:91)
at org.junit.Assert.assertTrue(Assert.java:43)
at org.apache.zookeeper.server.quorum.QuorumPeerMainTest.testBadPeerAddressInQuorum(QuorumPeerMainTest.java:434)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:52)
at org.junit.rules.TestWatchman$1.evaluate(TestWatchman.java:48)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:76)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184)
at org.junit.runners.ParentRunner.run(ParentRunner.java:236)
at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:39)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:518)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1052)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:906)
{noformat}
351401 No Perforce job exists for this issue. 2 351691
6 years, 2 weeks ago
Reviewed
0|i1ol5z:
ZooKeeper ZOOKEEPER-1773

incorrect reference to jline version/lib in docs

Bug Resolved Major Fixed Manikumar Patrick D. Hunt Patrick D. Hunt 01/Oct/13 17:20   04/Oct/13 07:07 03/Oct/13 12:57 3.5.0 3.5.0 documentation   0 3   The docs refer to an old version of jline

{noformat}
src/docs/src/documentation/content/xdocs/zookeeperAdmin.xml
227: <para><computeroutput>$ java -cp zookeeper.jar:lib/slf4j-api-1.6.1.jar:lib/slf4j-log4j12-1.6.1.jar:lib/log4j-1.2.15.jar:conf:src/java/lib/jline-0.9.94.jar \

src/docs/src/documentation/content/xdocs/zookeeperQuotas.xml
46: <para><computeroutput>$java -cp zookeeper.jar:src/java/lib/log4j-1.2.15.jar/conf:src/java/lib/jline-0.9.94.jar \
{noformat}
351377 No Perforce job exists for this issue. 1 351669
6 years, 24 weeks, 6 days ago
Reviewed
0|i1ol13:
ZooKeeper ZOOKEEPER-1772

config.guess downgraded from timestamp='2012-02-10' in 3.3.x to timestamp='2005-07-08' in 3.4.x

Bug Resolved Major Implemented Unassigned Chris Hall Chris Hall 01/Oct/13 09:43   01/Oct/13 17:57 01/Oct/13 17:57 3.4.5   c client   0 1   Was this intentional? If so, why? 351284 No Perforce job exists for this issue. 0 351576
6 years, 25 weeks, 2 days ago 0|i1okgf:
ZooKeeper ZOOKEEPER-1771

ZooInspector authentication

Improvement Closed Minor Fixed Benjamin Jaton Benjamin Jaton Benjamin Jaton 01/Oct/13 03:04   13/Mar/14 14:16 08/Oct/13 01:40 3.4.5, 3.5.0 3.4.6, 3.5.0 contrib   0 3   ZooInspector doesn't support authentication, so it always connects as anonymous to the ensemble.
It would be nice to be able to configure the authentication scheme+data in order to browse the nodes that have ACLs set.
351226 No Perforce job exists for this issue. 3 351518
6 years, 2 weeks ago Added authentication for ZooInspector.
Reviewed
0|i1ok3j:
ZooKeeper ZOOKEEPER-1770

NullPointerException in SnapshotFormatter

Bug Closed Minor Fixed Germán Blanco Germán Blanco Germán Blanco 30/Sep/13 04:00   13/Mar/14 14:16 01/Oct/13 19:46 3.4.5 3.4.6, 3.5.0     0 4   Windos, Java 1.7 SnapshotFormatter fails with a NullPointerException when parsing one snapshot (with "null" data in one Znode):
Exception in thread "main" java.lang.NullPointerException
at org.apache.zookeeper.server.SnapshotFormatter.printZnode(SnapshotFormatter.java:90)
at org.apache.zookeeper.server.SnapshotFormatter.printZnode(SnapshotFormatter.java:95)
at org.apache.zookeeper.server.SnapshotFormatter.printZnode(SnapshotFormatter.java:95)
at org.apache.zookeeper.server.SnapshotFormatter.printZnode(SnapshotFormatter.java:95)
at org.apache.zookeeper.server.SnapshotFormatter.printZnodeDetails(SnapshotFormatter.java:79)
at org.apache.zookeeper.server.SnapshotFormatter.printDetails(SnapshotFormatter.java:71)
at org.apache.zookeeper.server.SnapshotFormatter.run(SnapshotFormatter.java:67)
at org.apache.zookeeper.server.SnapshotFormatter.main(SnapshotFormatter.java:51)
351022 No Perforce job exists for this issue. 3 351314
6 years, 2 weeks ago
Reviewed
0|i1oiu7:
ZooKeeper ZOOKEEPER-1769

ZooInspector can't display node data/metadata/ACLs

Bug Resolved Minor Fixed Benjamin Jaton Benjamin Jaton Benjamin Jaton 29/Sep/13 13:12   01/Oct/13 07:11 01/Oct/13 01:56 3.5.0 3.5.0 contrib   0 4   Ubuntu There seem to be a few bugs in the trunk that prevent ZooInspector to load the node viewers ( the 3 tabs in the main windows when you select a ZK node in the tree don't show up any more ).
Apparently it has been introduced 2 years ago after a refactoring about icons and another about partially fixing a typo ("veiwer" -> "viewer").

Note: the bug is only in trunk, 3.4 is fine.
patch 350979 No Perforce job exists for this issue. 1 351270
6 years, 25 weeks, 2 days ago 0|i1oikf:
ZooKeeper ZOOKEEPER-1768

Cluster fails election loop until the device is full

Bug Open Major Unresolved Flavio Paiva Junqueira yuxin.yan yuxin.yan 27/Sep/13 05:19   05/Feb/20 07:16   3.4.5 3.7.0, 3.5.8 leaderElection   0 2   Hi,
I have a five nodes cluster versioned 3.4.5 and now i find one node is offline.
Firstly i restart the node but i find that "Error contacting service. It is probably not running." and i find that the node always elect the leader and always sync the snapshot logs and the device will be full every ten mins.
so could someone help me? i will put the log and zoo.cfg in the attachment.
Thanks all.
yyx,
350723 No Perforce job exists for this issue. 2 351014
6 years, 19 weeks, 1 day ago 0|i1ogzr:
ZooKeeper ZOOKEEPER-1767

DEBUG log statements use SLF4j {} format to improve performance

Improvement Patch Available Minor Unresolved Jackie Chang Jackie Chang Jackie Chang 26/Sep/13 21:28   22/Feb/15 17:31   3.4.5       0 3   As a coordination service, ZooKeeper is meant to be high performant. DEBUG logs are not normally viewed (see Doug Cutting's comment in HADOOP-953). I propose to add a conditional check to each DEBUG log stmt to improve performance.

Firstly, previous issues added a condition check before a DEBUG log stmt. For example, in ZOOKEEPER-558:
{code}
- LOG.debug("Got notification sessionid:0x"
- + Long.toHexString(sessionId));
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("Got notification sessionid:0x" +
+ Long.toHexString(sessionId));
+ }
{code}
And in ZOOKEEPER-259:
{code}
- LOG
- .debug("Got ping sessionid:0x"
- + Long.toHexString(sessionId));
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("Got ping response for sessionid:0x"
+ + Long.toHexString(sessionId)
+ + " after "
+ + ((System.nanoTime() - lastPingSentNs) / 1000000)
+ + "ms");
+ }
{code}
Secondly, its underlying cause is that:
* *If a DEBUG log stmt is unguarded, the string operations (most likely concatenations) are actually conducted even though the log event doesn't happen b/c a level less verbose than DEBUG is configured.*
* Adding the conditional check creates another basic block in Java bytecode. And instructions inside that basicblock is executed only when execution path goes into it. But this only happens when the path passes the test. Detailed explanations are in a StackOverflow thread: http://stackoverflow.com/questions/10428447/log-debug-enabled-check-in-java

An alternative solution is to move from log4j to slf4j and use the "{}" format. A workaround now is to add all conditional checks. The additional overhead is marginal (possibly compare-and-jump instruction(s) in Java bytecode) compared to saved computation of expensive string creations and concatenations.

Its counterpart in Hadoop has been accepted: HADOOP-6884.
350688 No Perforce job exists for this issue. 2 350979
5 years, 4 weeks, 4 days ago 0|i1ogrz:
ZooKeeper ZOOKEEPER-1766

Consistent log severity level guards and statements

Improvement Resolved Minor Fixed Jackie Chang Jackie Chang Jackie Chang 26/Sep/13 00:25   12/Feb/15 04:51 04/Oct/13 18:22 3.4.5 3.5.0     0 3   A log statement should be guarded by its matching severity level. A log statement like

if (LOG.isTraceEnabled()) {
LOG.info("Session closing: 0x" + Long.toHexString(sessionId));

doesn't make much sense because the log message is only printed out when TRACE-level is enabled. This inconsistency was possibly introduced when developers demoted the original log statement from INFO but forgot to change its corresponding log severity level.
350463 No Perforce job exists for this issue. 1 350756
6 years, 24 weeks, 5 days ago 0|i1ofen:
ZooKeeper ZOOKEEPER-1765

Update code conventions link on "How to contribute" page

Bug Closed Trivial Fixed Patrick D. Hunt Flavio Paiva Junqueira Flavio Paiva Junqueira 25/Sep/13 18:21   13/Mar/14 14:17 01/Oct/13 20:03   3.4.6, 3.5.0 documentation   0 2   350420 No Perforce job exists for this issue. 0 350713
6 years, 2 weeks ago
Reviewed
0|i1of53:
ZooKeeper ZOOKEEPER-1764

ZooKeeper attempts at SASL eventhough it shouldn't

Bug Closed Major Implemented Unassigned Kuba Skopal Kuba Skopal 25/Sep/13 05:51   13/Mar/14 14:17 25/Sep/13 13:05 3.4.4 3.4.6 java client   0 3   We are using a proprietary SASL solution, but we don't want to use it with ZooKeeper. Unfortunately it seems, that there is no way to disable SASL for ZooKeeper as the code only checks for presence of "java.security.auth.login.config" system property to determine whether SASL should be used or not.
For us it means, that ZooKeeper client just shuts down after SASL is initialized. What happens:

1) System.getProperty("java.security.auth.login.config") is initially null
2) ZooKeeper is initialized and used
3) Our JAAS/SASL component is initialized
4) System.getProperty("java.security.auth.login.config") is not null anymore
5) ZooKeeperSaslClient.clientTunneledAuthenticationInProgress() suddenly picks up the new property and starts returning true
6) ClientCnxnSocketNIO.findSendablePacket() suddenly stops returning any packets since clientTunneledAuthenticationInProgress is always true

The communication is halted and eventually times out.
350277 No Perforce job exists for this issue. 0 350570
6 years, 2 weeks ago Solved by ZOOKEEPER-1657 0|i1oe9j:
ZooKeeper ZOOKEEPER-1763

Upgrade the netty version

Improvement Resolved Minor Duplicate Nicolas Liochon Nicolas Liochon Nicolas Liochon 23/Sep/13 08:14   11/Oct/13 17:32 11/Oct/13 17:32 3.4.5       0 3   2 years ago (in https://github.com/netty/netty/issues/103), Netty changed their group-id from org.jboss.netty to io.netty. ZooKeeper is still on the 3.2.5, so applications using 3.3 cannot use the maven "dependencyManagement" feature, as the group id differ. HBase & Hadoop 2 are on the branch 3.3+, with the new group id.

Note that the netty 4 changes the package name as well. That's not the case for Netty 3.3+.
349870 No Perforce job exists for this issue. 1 350168
6 years, 23 weeks, 6 days ago 0|i1obs7:
ZooKeeper ZOOKEEPER-1762

ZOOKEEPER-1760 Implement 'check' version cli command

Sub-task Resolved Major Won't Fix Unassigned Rakesh Radhakrishnan Rakesh Radhakrishnan 22/Sep/13 20:01   25/Sep/13 05:43 25/Sep/13 05:43   3.5.0 java client   0 1   349777 No Perforce job exists for this issue. 1 350075
6 years, 26 weeks, 1 day ago 0|i1ob7j:
ZooKeeper ZOOKEEPER-1761

ZOOKEEPER-1760 Expose 'check' version api in ZooKeeper client

Sub-task Resolved Major Won't Fix Rakesh Radhakrishnan Rakesh Radhakrishnan Rakesh Radhakrishnan 22/Sep/13 19:05   25/Sep/13 05:42 25/Sep/13 05:42   3.5.0 java client   0 3   Implement ZooKeeper#check api 349772 No Perforce job exists for this issue. 1 350070
6 years, 26 weeks, 1 day ago 0|i1ob6f:
ZooKeeper ZOOKEEPER-1760

Provide an interface for check version of a node

New Feature Resolved Major Won't Fix Rakesh Radhakrishnan Rakesh Radhakrishnan Rakesh Radhakrishnan 22/Sep/13 19:02   25/Sep/13 05:44 25/Sep/13 04:26   3.5.0 java client   0 6   ZOOKEEPER-1761, ZOOKEEPER-1762 The idea of this JIRA is to discuss the check version interface which is used to see the existence of a node for the specified version. Presently only multi transaction api has this interface, this umbrella JIRA is to make 'check version' api part of ZooKeeper# main apis and cli command.
349771 No Perforce job exists for this issue. 0 350069
6 years, 26 weeks, 1 day ago 0|i1ob67:
ZooKeeper ZOOKEEPER-1759

Adding ability to allow READ operations for authenticated users, versus keeping ACLs wide open for READ

Improvement Resolved Major Fixed Yuliya Feldman Yuliya Feldman Yuliya Feldman 18/Sep/13 02:28   12/Feb/15 08:44 29/Sep/13 19:47 3.4.5 3.5.0 server   0 6   Java, SASL authentication, security Today when using SASLAuthenticationProvider to authenticate Zookeeper Clients access to the data based on ACLS set on znodes there is no other choice but to set READ ACLs to be "world", "anyone" with the way how
{code:java}
public boolean matches(String id,String aclExpr)
{code}
is currently implemented. It means that any unauthenticated user can read the data when application needs to make sure that not only creator of a znode can read the content.
Proposal is to introduce new property: "zookeeper.readUser" that if incoming id matches to the value of that property it will be allowed to proceed in "match" method.
So creator of a znode instead of
{code:java}
ACL acl1 = new ACL(Perms.ADMIN | Perms.CREATE | Perms.WRITE | Perms.DELETE, Ids.AUTH_IDS);
ACL acl2 = new ACL(Perms.READ, Ids.ANYONE_ID_UNSAFE);
{code}
will need to do
{code:java}
ACL acl1 = new ACL(Perms.ADMIN | Perms.CREATE | Perms.WRITE | Perms.DELETE, Ids.AUTH_IDS);
ACL acl2 = new ACL(Perms.READ, new Id("sasl", "anyone"));
{code}
Assuming that value of "zookeeper.readUser" property was "anyone".
This way at least READ access on corresponding znode has to be authenticated.
349122 No Perforce job exists for this issue. 7 349420
6 years, 25 weeks, 4 days ago 0|i1o75z:
ZooKeeper ZOOKEEPER-1758

Add documentation for zookeeper.observer.syncEnabled flag

Improvement Closed Minor Fixed Thawan Kooburat Thawan Kooburat Thawan Kooburat 13/Sep/13 20:43   13/Mar/14 14:17 30/Sep/13 16:55   3.4.6, 3.5.0     0 3   348535 No Perforce job exists for this issue. 2 348832
6 years, 2 weeks ago 0|i1o3jj:
ZooKeeper ZOOKEEPER-1757

Adler32 may not be sufficient to protect against data corruption

Bug Open Minor Unresolved Unassigned Thawan Kooburat Thawan Kooburat 12/Sep/13 19:57   13/Sep/13 22:04       server   0 5   Linux. Oracle JDK6/7 I was investigating data inconsistency bug in our internal branch. One possible area is snapshot/txnlog corruption. So I wrote a more robust corruption test and found that it is easy to break our checksum algorithm which is Adler32.

When this happen, it is more likely that corrupted data will fail other sanity check during deserialization phase, but it is still scary that it can pass the checksum.
348317 No Perforce job exists for this issue. 2 348613
6 years, 27 weeks, 5 days ago 0|i1o273:
ZooKeeper ZOOKEEPER-1756

zookeeper_interest() in C client can return a timeval of 0

Bug Closed Major Fixed Eric Lindvall Eric Lindvall Eric Lindvall 08/Sep/13 00:45   13/Mar/14 14:17 16/Dec/13 16:50 3.3.4, 3.4.5 3.4.6, 3.5.0 c client   0 5   If the client is connected to a zookeeper server that has hung while there is an outstanding request, zookeeper_interest() can return a timeval of 0 because send_to will be negative. 347409 No Perforce job exists for this issue. 5 347708
6 years, 2 weeks ago 0|i1nwmf:
ZooKeeper ZOOKEEPER-1755

Concurrent operations of four letter 'dump' ephemeral command and killSession causing NPE

Bug Closed Major Fixed Rakesh Radhakrishnan Rakesh Radhakrishnan Rakesh Radhakrishnan 07/Sep/13 14:12   13/Mar/14 14:17 18/Feb/14 20:38 3.4.5 3.4.6, 3.5.0 server   0 9   Potential problem occurs, when executing four letter 'dump' command and at the meantime zkserver has triggered session closure and removing the related information from the DataTree.

Please see the exception:
{code}
java.lang.NullPointerException
at org.apache.zookeeper.server.DataTree.dumpEphemerals(DataTree.java:1278)
at org.apache.zookeeper.server.DataTreeTest$1.run(DataTreeTest.java:82)
{code}
347393 No Perforce job exists for this issue. 3 347692
6 years, 2 weeks ago 0|i1nwiv:
ZooKeeper ZOOKEEPER-1754

Read-only server allows to create znode

Bug Closed Critical Fixed Rakesh Radhakrishnan Rakesh Radhakrishnan Rakesh Radhakrishnan 06/Sep/13 11:42   13/Mar/14 14:17 17/Sep/13 19:14 3.5.0 3.4.6, 3.5.0 server   0 7   347281 No Perforce job exists for this issue. 4 347580
6 years, 2 weeks ago 0|i1nvu7:
ZooKeeper ZOOKEEPER-1753

ClientCnxn is not properly releasing the resources, which are used to ping RwServer

Bug Closed Major Fixed Rakesh Radhakrishnan Rakesh Radhakrishnan Rakesh Radhakrishnan 05/Sep/13 10:37   13/Mar/14 14:17 18/Sep/13 09:15   3.4.6, 3.5.0 java client   0 7   While pinging to the RwServer, ClientCnxn is opening a socket and using BufferedReader. These are not properly closed in finally block and could cause leaks on exceptional cases.

ClientCnxn#pingRwServer()
{code}
try {
Socket sock = new Socket(addr.getHostName(), addr.getPort());
BufferedReader br = new BufferedReader(
new InputStreamReader(sock.getInputStream()));
......
sock.close();
br.close();
} catch (ConnectException e) {
// ignore, this just means server is not up
} catch (IOException e) {
// some unexpected error, warn about it
LOG.warn("Exception while seeking for r/w server " +
e.getMessage(), e);
}
{code}
347054 No Perforce job exists for this issue. 3 347353
6 years, 2 weeks ago 0|i1nufr:
ZooKeeper ZOOKEEPER-1752

Download area contains obsolete releases

Bug Resolved Major Fixed Patrick D. Hunt Sebb Sebb 01/Sep/13 19:55   03/Mar/19 18:33 03/Mar/19 12:50         0 2   The download area under http://www.apache.org/dist/zookeeper/ contains several superseded releases.

It is important that only the latest release of each currently maintained product line is stored on the main ASF mirrors. Links to older releases can be provided on download pages, but the links should be to the ASF archive server http://archive.apache.org/dist/zookeeper/

See http://www.apache.org/dev/release.html#when-to-archive
346456 No Perforce job exists for this issue. 0 346757
1 year, 2 weeks, 4 days ago 0|i1nqrj:
ZooKeeper ZOOKEEPER-1751

ClientCnxn#run could miss the second ping or connection get dropped before a ping

Bug Closed Major Fixed Jeffrey Zhong Jeffrey Zhong Jeffrey Zhong 30/Aug/13 20:56   13/Mar/14 14:17 17/Sep/13 22:08 3.3.5, 3.4.5 3.4.6, 3.5.0     0 7   We could throw SessionTimeoutException exception even when timeToNextPing may also be negative depending on the time when the following line is executed by the thread because we check time out before sending a ping.
{code}
to = readTimeout - clientCnxnSocket.getIdleRecv();
{code}

In addition, we only ping twice no matter how long the session time out value is. For example, we set session time out = 60mins then we only try ping twice in 40mins window. Therefore, the connection could be dropped by OS after idle time out.

The issue is causing randomly "connection loss" or "session expired" issues in client side which is bad for applications like HBase.
346375 No Perforce job exists for this issue. 1 346676
6 years, 2 weeks ago 0|i1nq9j:
ZooKeeper ZOOKEEPER-1750

Race condition producing NPE in NIOServerCnxn.toString

Bug Closed Minor Fixed Rakesh Radhakrishnan Helen Hastings Helen Hastings 30/Aug/13 18:57   13/Mar/14 14:16 04/Oct/13 19:16 3.5.0 3.4.6, 3.5.0 server   0 9   The socket is closed and the variable "sock" is set to null for normal reasons, but the toString method is called before "sock" can be set again, producing a NullPointerException.

Stack trace:

2013-08-29 01:49:19,991 6277 [CommitProcWorkThread-3] WARN org.apache.zookeeper.server.WorkerService - Unexpected exception
java.lang.NullPointerException
at org.apach.zookeeper.server.NIOServerCnxn.toString(NIOServerCnxn.java:961)
at java.lang.String.valueOf(String.java:2854)
at java.lang.StringBuilder.append(StringBuilder.java:128)
at org.apache.zookeeper.server.NIOServerCnxn.process(NIOServerCnxn.java:1104)
at org.apache.zookeeper.server.WatchManager.triggerWatch(WatchManager.java:120)
at org.apache.zookeeper.server.WatchManager.triggerWatch(WatchManager.java:92)
at org.apache.zookeeper.server.DataTree.createNode(DataTree.java:544)
at org.apache.zookeeper.server.DataTree.processTxn(DataTree.java:805)
at org.apache.zookeeper.server.ZKDatabase.processTxn(ZKDatabase.java:319)
at org.apache.zookeeper.server.ZooKeeperServer.processTxn(ZooKeeperServer.java:967)
at org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:115)
at org.apache.zookeeper.server.quorum.Leader$ToBeAppliedRequestProcessor.processRequest(Leader.java:859)
at org.apache.zookeeper.server.quorum.CommitProcessor$CommitWorkRequest.doWork(CommitProcessor.java:271)
at org.apache.zookeeper.server.WorkerService$ScheduledWorkRequest.run(WorkerService.java:152)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)
346355 No Perforce job exists for this issue. 4 346656
6 years, 2 weeks ago 0|i1nq53:
ZooKeeper ZOOKEEPER-1749

Login outside of Zookeeper client

Improvement Open Major Unresolved Kai Zheng Kai Zheng Kai Zheng 30/Aug/13 18:21   05/Feb/20 07:16     3.7.0, 3.5.8 java client   0 4   This proposes to allow Zookeeper client to reuse login credentials and subject from outside, avoiding redundant logins and related configurations for services that utilizes Zookeeper. 346350 No Perforce job exists for this issue. 2 346651
6 years, 4 days ago 0|i1nq3z:
ZooKeeper ZOOKEEPER-1748

TCP keepalive for leader election connections

Improvement Resolved Minor Fixed Ben Sherman Antal Sasvári Antal Sasvári 29/Aug/13 10:57   22/Oct/19 18:00 08/Jun/17 11:42 3.4.5, 3.5.0 3.4.11, 3.5.4, 3.6.0 leaderElection   8 25   Linux, Java 1.7 In our system we encountered the following problem:

If the system is stable, and there is no leader election, the leader election port connections are open for very long time without any packets being sent on them.
Some network elements silently drop the established TCP connection after a timeout if there are no packets being sent on it. In this case the ZK servers will not notice the connection loss. This causes additional delay later when the next leader election is started, as the TCP connections are not alive any more.

We would like to be able to enable TCP keepalive on the leader election sockets in order to prevent the connection timeout in some network elements due to connection inactivity.
This could be controlled by adding a new config parameter called tcpKeepAlive in the ZooKeeper configuration file. It would be only applicable in case of algorithm 3 (TCP based fast leader election), having the default value false.

If tcpKeepAlive is set to true, the TCP keepalive flag should be enabled for the leader election sockets in QuorumCnxManager.setSockOpts() by calling sock.setKeepAlive(true).
We have tested this change successfully in our environment.

Please comment whether you see any problem with this. If not, I am going to submit a patch.

I've been told that e.g. Apache ActiveMQ also has a config option for similar purpose called transport.keepalive.
346080 No Perforce job exists for this issue. 1 346381
21 weeks, 2 days ago 0|i1nogn:
ZooKeeper ZOOKEEPER-1747

Zookeeper server fails to start if transaction log file is corrupted

Bug Resolved Major Duplicate Unassigned Sergey Maslyakov Sergey Maslyakov 29/Aug/13 10:04   12/Sep/13 11:06 12/Sep/13 11:05 3.4.5   server   1 2   Solaris10/x86, Java 1.6 On multiple occasions when ZK was not able to write out a transaction log or a snapshot file, the consequent attempt to restart the server fails. Usually it happens when the underlying file system filled up; thus, preventing ZK server from writing out consistent data file.

Upon start-up, the server reads in the snapshot and the transaction log. If the deserializer fails and throws an exception, server terminates. Please see the stack trace below.

Server not coming up for whatever reason is often an undesirable condition. It would be nice to have an option to force-ignore parsing errors, especially, in the transaction log. A check sum on the data could be a possible solution to ensure the integrity and "parsability".

Another robustness enhancement could be via proper handling of the condition when snapshot or transaction log cannot be completely written to disk. Basically, better handling of write errors.


{noformat}
2013-08-28 12:05:30,732 ERROR [ZooKeeperServerMain] Unexpected exception, exiting abnormally
java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:375)
at org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63)
at org.apache.zookeeper.server.persistence.FileHeader.deserialize(FileHeader.java:64)
at org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.inStreamCreated(FileTxnLog.java:558)
at org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.createInputArchive(FileTxnLog.java:577)
at org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.goToNextLog(FileTxnLog.java:543)
at org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.next(FileTxnLog.java:625)
at org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:160)
at org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:223)
at org.apache.zookeeper.server.ZooKeeperServer.loadData(ZooKeeperServer.java:250)
at org.apache.zookeeper.server.ZooKeeperServer.startdata(ZooKeeperServer.java:383)
at org.apache.zookeeper.server.NIOServerCnxnFactory.startup(NIOServerCnxnFactory.java:122)
at org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:112)
at org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:86)
at org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:52)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:129)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:78)
{noformat}
346076 No Perforce job exists for this issue. 0 346377
6 years, 28 weeks ago 0|i1nofr:
ZooKeeper ZOOKEEPER-1746

AsyncCallback.*Callback don't have any Javadoc

Improvement Resolved Major Fixed Hongchao Deng Tsuyoshi Ozawa Tsuyoshi Ozawa 28/Aug/13 15:18   26/Jun/14 07:19 25/Jun/14 13:09 3.4.6 3.4.7, 3.5.0 documentation   0 4   AsyncCallback.*Callback don't have any Javadoc. This forces users to read source code or sample code to understand what their arguments stand for or how one is difference from the others. newbie 345927 No Perforce job exists for this issue. 2 346228
5 years, 39 weeks ago
Reviewed
0|i1nnin:
ZooKeeper ZOOKEEPER-1745

Wrong Import-Package in the META-INF/MANIFEST.MF of zookeeper 3.4.5 bundle

Bug Resolved Major Fixed Jean-Baptiste Onofré Xilai Dai Xilai Dai 26/Aug/13 05:25   24/Apr/14 21:58 24/Apr/14 21:58 3.4.5 3.4.6, 3.5.0 server   0 6   Java 7 Import-Package: javax.management,org.apache.log4j,org.osgi.framework;v
ersion="[1.4,2.0)",org.osgi.util.tracker;version="[1.1,2.0)"

the "org.apache.log4j" should be replaced by "org.slf4j", because from the source codes, zookeeper server classes import org.slf4j.* for logging.

currently will get:

Caused by: java.lang.NoClassDefFoundError: org/slf4j/LoggerFactory
at org.apache.zookeeper.server.quorum.QuorumPeerConfig.<clinit>(QuorumPeerConfig.java:46)

when try to create instance for some of its classes in OSGi container (e.g. apache karaf)
345416 No Perforce job exists for this issue. 0 345717
5 years, 47 weeks, 6 days ago 0|i1nkd3:
ZooKeeper ZOOKEEPER-1744

clientPortAddress breaks "zkServer.sh status"

Bug Closed Critical Fixed Nick Ohanian Nick Ohanian Nick Ohanian 22/Aug/13 16:23   13/Mar/14 14:17 24/Oct/13 01:12 3.4.5 3.4.6, 3.5.0 scripts   0 6   When "clientPortAddress" is used in the config file (zoo.cfg), zkServer.sh's status command runs a grep command that matches both "clientPort" and "clientPortAddress". This creates an extra argument for FourLetterWordMain, which fails, so the status command incorrectly indicates that it couldn't connect to the server.

Also, "localhost" is hardcoded as the target host for FourLetterWordMain. The "clientPortAddress" should be used if it is provided in the config file.
345041 No Perforce job exists for this issue. 2 345342
6 years, 2 weeks ago
Reviewed
0|i1ni27:
ZooKeeper ZOOKEEPER-1743

Chocolatey package for automating Zookeeper installation in Windows

Improvement Resolved Minor Won't Fix Unassigned Andrew Pennebaker Andrew Pennebaker 21/Aug/13 12:43   03/Mar/16 19:35 03/Mar/16 19:35     build   0 1   Chocolatey (http://chocolatey.org/)
Windows XP+
Ubuntu has "apt-get install zookeeper", simplifying installation in Linux. Similarly, Mac has "brew install zookeeper". Could we help out our Windows users by submitting a Chocolatey package? 344812 No Perforce job exists for this issue. 0 345112
4 years, 3 weeks ago 0|i1ngn3:
Generated at Fri Mar 20 00:35:51 UTC 2020 by Song Xu using Jira 8.3.4#803005-sha1:1f96e09b3c60279a408a2ae47be3c745f571388b.